2024-03-04T15:14:45.843 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T15:14:45.843 DEBUG:teuthology.run:Teuthology command: teuthology --name tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi --archive /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170 --owner scheduled_tonay@teuthology --description orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} --verbose -- /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/orig.config.yaml 2024-03-04T15:14:45.871 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:14:45.955 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170 branch: wip-asm0deuz-testing-2024-03-04-1211 description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} email: tonay@ibm.com first_in_suite: false job_id: '7581170' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-asm0deuz-testing-2024-03-04-1211 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-asm0deuz-testing-2024-03-04-1211 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 owner: scheduled_tonay@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 3905 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-asm0deuz-testing-2024-03-04-1211 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAtfG8/IeHVzu1kUVyWoGuv8wn8we0PiWl4Kdr7CreaPLybqYO9k3EL0GF557B7CYgh35UBoTsav0uW1y4Rvct8= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKIQpmCis09si6l5fmDFPlVYx2CB2vjFirqGOyXiKNVGookbxjrDHPn21w/9SWtyGQu31tsgXKqiuOOTtJvOJvU= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFOHYWIpXU+ozeWSPqmBmSb9SYag2Eyafp0sZCWWf2UTe8dQXltW6tzct8gGYZm/i1Tn2jOGr/gJNm7FXDTc5M= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_15:07:38 tube: smithi user: tonay verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T15:14:45.956 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa; will attempt to use it 2024-03-04T15:14:45.956 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks 2024-03-04T15:14:45.956 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T15:14:45.958 INFO:teuthology.task.internal:Checking packages... 2024-03-04T15:14:45.981 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9ca0b5de35ed230e84387abdd8e619ed8e4f3282' 2024-03-04T15:14:45.981 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T15:14:45.981 INFO:teuthology.packaging:ref: None 2024-03-04T15:14:45.981 INFO:teuthology.packaging:tag: None 2024-03-04T15:14:45.981 INFO:teuthology.packaging:branch: wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:14:45.981 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:14:45.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:14:46.206 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1826.g9ca0b5de 2024-03-04T15:14:46.208 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T15:14:46.214 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T15:14:46.214 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T15:14:46.227 INFO:teuthology.task.internal:Saving configuration 2024-03-04T15:14:46.239 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T15:14:46.250 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:14:46.285 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:08:57.467354', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAtfG8/IeHVzu1kUVyWoGuv8wn8we0PiWl4Kdr7CreaPLybqYO9k3EL0GF557B7CYgh35UBoTsav0uW1y4Rvct8='} 2024-03-04T15:14:46.316 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:08:57.469301', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKIQpmCis09si6l5fmDFPlVYx2CB2vjFirqGOyXiKNVGookbxjrDHPn21w/9SWtyGQu31tsgXKqiuOOTtJvOJvU='} 2024-03-04T15:14:46.349 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:08:57.470422', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFOHYWIpXU+ozeWSPqmBmSb9SYag2Eyafp0sZCWWf2UTe8dQXltW6tzct8gGYZm/i1Tn2jOGr/gJNm7FXDTc5M='} 2024-03-04T15:14:46.349 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T15:14:46.360 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-04T15:14:46.360 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-04T15:14:46.360 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-04T15:14:46.360 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T15:14:46.514 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9e9add0f70>, signals=[15]) 2024-03-04T15:14:46.514 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T15:14:46.525 INFO:teuthology.task.internal:Opening connections... 2024-03-04T15:14:46.525 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-03-04T15:14:46.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:14:46.636 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T15:14:46.637 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:14:46.747 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-03-04T15:14:46.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:14:46.842 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T15:14:46.852 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-03-04T15:14:46.870 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-03-04T15:14:46.871 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-03-04T15:14:46.927 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:14:46.928 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:14:46.929 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-04T15:14:46.977 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-03-04T15:14:46.995 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-03-04T15:14:46.995 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-03-04T15:14:47.053 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:14:47.054 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:14:47.054 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:14:47.054 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:14:47.054 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:14:47.054 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:14:47.054 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-03-04T15:14:47.077 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-03-04T15:14:47.094 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-03-04T15:14:47.094 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="CentOS Stream" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="9" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:ID="centos" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="9" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31" 2024-03-04T15:14:47.151 INFO:teuthology.orchestra.run.smithi160.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:14:47.152 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:14:47.152 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:14:47.152 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:14:47.152 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:14:47.152 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:14:47.152 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-03-04T15:14:47.177 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T15:14:47.188 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T15:14:47.203 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T15:14:47.204 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:14:47.206 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:14:47.208 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:14:47.224 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T15:14:47.243 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T15:14:47.243 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:14:47.263 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:14:47.266 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:14:47.294 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T15:14:47.398 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T15:14:47.411 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T15:14:47.411 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T15:14:47.411 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:14:47.412 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:14:47.412 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:14:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:14:47.429 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-03-04T15:14:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:14:47.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:14:47.431 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-04T15:14:47.431 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-04T15:14:47.486 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:14:47.486 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.14.0-361.el9.x86_64 2024-03-04T15:14:47.486 DEBUG:teuthology.orchestra.run.smithi160:> sudo yum install -y kernel 2024-03-04T15:14:47.488 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:14:47.488 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-03-04T15:14:47.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-03-04T15:14:47.489 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:14:47.489 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-361.el9.x86_64 2024-03-04T15:14:47.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-03-04T15:14:49.631 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T15:14:49.757 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-04T15:14:49.769 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-04T15:14:51.273 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-04T15:14:51.316 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-04T15:14:51.361 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-04T15:14:52.162 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-04T15:14:52.300 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-04T15:14:52.359 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-04T15:14:54.173 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-04T15:14:57.744 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-04T15:14:58.118 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-04T15:14:59.616 INFO:teuthology.orchestra.run.smithi160.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-03-04T15:15:01.245 INFO:teuthology.orchestra.run.smithi160.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-03-04T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-04T15:15:03.495 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 15 kB 00:00 2024-03-04T15:15:05.941 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 20 MB 00:02 2024-03-04T15:15:06.200 INFO:teuthology.orchestra.run.smithi160.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-04T15:15:08.416 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:15:08.416 INFO:teuthology.orchestra.run.smithi160.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:15:08.498 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-03-04T15:15:08.506 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-03-04T15:15:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-03-04T15:15:08.506 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-03-04T15:15:08.506 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-03-04T15:15:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout:Installing dependencies: 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout:Install 4 Packages 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 100 M 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 123 M 2024-03-04T15:15:08.507 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-03-04T15:15:08.827 INFO:teuthology.orchestra.run.smithi160.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-04T15:15:09.086 INFO:teuthology.orchestra.run.smithi160.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 39 MB/s | 21 MB 00:00 2024-03-04T15:15:09.503 INFO:teuthology.orchestra.run.smithi160.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 41 MB/s | 39 MB 00:00 2024-03-04T15:15:09.820 INFO:teuthology.orchestra.run.smithi160.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 34 MB/s | 33 MB 00:00 2024-03-04T15:15:09.823 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:15:09.823 INFO:teuthology.orchestra.run.smithi160.stdout:Total 76 MB/s | 100 MB 00:01 2024-03-04T15:15:10.712 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-03-04T15:15:10.997 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-04T15:15:11.111 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-03-04T15:15:11.111 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-03-04T15:15:12.477 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-03-04T15:15:12.477 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-03-04T15:15:13.362 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:15:13.362 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:15:13.452 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-04T15:15:13.459 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-04T15:15:13.459 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-04T15:15:13.460 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-03-04T15:15:13.461 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:15:13.461 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 100 M 2024-03-04T15:15:13.461 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 123 M 2024-03-04T15:15:13.461 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-04T15:15:14.015 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-04T15:15:14.332 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 35 MB/s | 21 MB 00:00 2024-03-04T15:15:14.688 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 926 kB/s | 19 MB 00:21 2024-03-04T15:15:14.740 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 39 MB/s | 39 MB 00:01 2024-03-04T15:15:15.049 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 32 MB/s | 33 MB 00:01 2024-03-04T15:15:15.052 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:15:15.052 INFO:teuthology.orchestra.run.smithi053.stdout:Total 63 MB/s | 100 MB 00:01 2024-03-04T15:15:15.197 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-03-04T15:15:15.862 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:15:15.978 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-04T15:15:15.994 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:15:16.433 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-04T15:15:16.433 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-04T15:15:17.849 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-04T15:15:17.849 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-04T15:15:18.839 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:15:19.022 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:15:19.192 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-04T15:15:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-04T15:15:21.073 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-03-04T15:15:21.407 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:15:21.558 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:15:22.933 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-03-04T15:15:23.751 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:15:24.057 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:24.512 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:15:24.720 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:15:28.016 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-03-04T15:15:28.823 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:29.819 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:15:29.940 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:30.339 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:15:30.340 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:15:30.440 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T15:15:30.446 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-03-04T15:15:30.447 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:15:30.448 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 100 M 2024-03-04T15:15:30.448 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 123 M 2024-03-04T15:15:30.448 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T15:15:30.767 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-04T15:15:31.035 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 39 MB/s | 21 MB 00:00 2024-03-04T15:15:31.418 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 52 MB/s | 33 MB 00:00 2024-03-04T15:15:31.826 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 30 MB/s | 39 MB 00:01 2024-03-04T15:15:31.829 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:15:31.830 INFO:teuthology.orchestra.run.smithi064.stdout:Total 72 MB/s | 100 MB 00:01 2024-03-04T15:15:32.750 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T15:15:33.198 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T15:15:33.199 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T15:15:34.634 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T15:15:34.634 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T15:15:35.061 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:37.515 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T15:15:38.201 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:15:38.335 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:15:41.286 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:15:41.469 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:15:46.603 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:15:46.731 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:51.897 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:58.142 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:58.142 INFO:teuthology.orchestra.run.smithi160.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:15:58.143 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:15:58.226 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:15:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:15:58.654 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:15:59.115 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-03-04T15:15:59.653 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:15:59.810 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:15:59.810 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:15:59.810 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:15:59.810 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:15:59.810 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:15:59.810 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-04T15:15:59.810 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:15:59.811 DEBUG:teuthology.orchestra.run.smithi160:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:15:59.931 INFO:teuthology.orchestra.run.smithi160.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:15:59.931 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:15:59.931 DEBUG:teuthology.orchestra.run.smithi160:> sudo rpm -qi grub2-tools 2024-03-04T15:15:59.964 INFO:teuthology.orchestra.run.smithi160.stdout:Name : grub2-tools 2024-03-04T15:15:59.964 INFO:teuthology.orchestra.run.smithi160.stdout:Epoch : 1 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Version : 2.06 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Release : 68.el9 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: x86_64 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Group : Unspecified 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Size : 8264254 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:License : GPLv3+ 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Packager : builder@centos.org 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Vendor : CentOS 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Summary : Support tools for GRUB. 2024-03-04T15:15:59.965 INFO:teuthology.orchestra.run.smithi160.stdout:Description : 2024-03-04T15:15:59.966 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:15:59.966 INFO:teuthology.orchestra.run.smithi160.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:15:59.966 INFO:teuthology.orchestra.run.smithi160.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:15:59.966 INFO:teuthology.orchestra.run.smithi160.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:15:59.966 INFO:teuthology.orchestra.run.smithi160.stdout:hardware devices. 2024-03-04T15:15:59.966 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:15:59.966 INFO:teuthology.orchestra.run.smithi160.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:15:59.967 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:15:59.967 INFO:teuthology.task.kernel:Updating grub on smithi160 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:15:59.967 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:16:00.675 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2024-03-04T15:16:02.001 INFO:teuthology.orchestra.run.smithi160.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:16:02.042 INFO:teuthology.orchestra.run.smithi160.stderr:done 2024-03-04T15:16:02.044 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2024-03-04T15:16:02.059 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.SzsM8Y0j1m 2024-03-04T15:16:02.059 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SzsM8Y0j1m 2024-03-04T15:16:02.127 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.SzsM8Y0j1m 2024-03-04T15:16:02.273 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.SzsM8Y0j1m is 6KB 2024-03-04T15:16:02.321 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.SzsM8Y0j1m 2024-03-04T15:16:02.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:16:02.404 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:16:02.404 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:16:02.404 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:16:02.404 INFO:teuthology.orchestra.run.smithi160.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:16:02.405 DEBUG:teuthology.orchestra.run.smithi160:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:16:02.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2024-03-04T15:16:05.557 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:05.557 INFO:teuthology.orchestra.run.smithi053.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:16:05.558 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:16:05.644 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:06.087 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:06.087 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:16:06.087 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:16:06.087 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:16:06.549 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:16:06.550 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-04T15:16:07.033 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:16:07.179 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:16:07.180 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:16:07.180 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:16:07.180 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:16:07.180 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:16:07.180 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-04T15:16:07.180 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:16:07.180 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:16:07.306 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:16:07.306 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:16:07.307 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-03-04T15:16:07.347 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-03-04T15:16:07.347 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-03-04T15:16:07.347 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 68.el9 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8264254 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-03-04T15:16:07.348 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-03-04T15:16:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:16:07.349 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:16:07.349 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:16:07.349 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:16:07.349 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-03-04T15:16:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:16:07.349 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:16:07.350 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:16:07.350 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:16:07.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:16:07.943 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-03-04T15:16:09.392 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:16:09.436 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-03-04T15:16:09.438 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-03-04T15:16:09.453 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.k6xwT6Ktch 2024-03-04T15:16:09.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.k6xwT6Ktch 2024-03-04T15:16:09.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.k6xwT6Ktch 2024-03-04T15:16:09.672 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.k6xwT6Ktch is 6KB 2024-03-04T15:16:09.720 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.k6xwT6Ktch 2024-03-04T15:16:09.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:16:09.813 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:16:09.813 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:16:09.813 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:16:09.813 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:16:09.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:16:09.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-03-04T15:16:22.778 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:22.779 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:16:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:16:22.865 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:23.340 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:23.340 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:16:23.341 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:16:23.341 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:16:23.800 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:16:23.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:16:23.800 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T15:16:23.800 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:16:23.800 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:16:23.800 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:16:23.801 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:16:23.801 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:16:23.801 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T15:16:24.283 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:16:24.427 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:16:24.427 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:16:24.427 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:16:24.427 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:16:24.428 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:16:24.428 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-04T15:16:24.428 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:16:24.428 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:16:24.554 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:16:24.554 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:16:24.554 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-03-04T15:16:24.591 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-03-04T15:16:24.591 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-03-04T15:16:24.591 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-03-04T15:16:24.591 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-03-04T15:16:24.591 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-03-04T15:16:24.592 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:16:24.593 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:16:24.593 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:16:24.593 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:16:24.593 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-03-04T15:16:24.593 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:16:24.593 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:16:24.594 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:16:24.594 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:16:24.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:16:25.236 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-03-04T15:16:26.737 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:16:26.758 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-03-04T15:16:26.761 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-03-04T15:16:26.775 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.rOWUoHqeyA 2024-03-04T15:16:26.776 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rOWUoHqeyA 2024-03-04T15:16:26.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.rOWUoHqeyA 2024-03-04T15:16:26.993 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.rOWUoHqeyA is 6KB 2024-03-04T15:16:27.042 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.rOWUoHqeyA 2024-03-04T15:16:27.057 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:16:27.125 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:16:27.125 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:16:27.125 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:16:27.125 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:16:27.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:16:27.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-03-04T15:16:32.560 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-04T15:16:32.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-03-04T15:16:32.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:16:39.981 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-04T15:16:39.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-04T15:16:39.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:16:51.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-03-04T15:16:57.276 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-04T15:16:57.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T15:16:57.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:16:58.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-04T15:17:00.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-03-04T15:17:00.136 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:17:03.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-03-04T15:17:07.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-04T15:17:07.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:17:15.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-03-04T15:17:15.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:17:15.708 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-04T15:17:24.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T15:17:24.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:17:25.948 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-03-04T15:17:27.836 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-04T15:17:37.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-04T15:17:37.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:17:38.506 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-04T15:17:38.870 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-04T15:17:38.870 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:17:38.870 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-04T15:17:38.884 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:17:38.885 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:17:38.885 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:17:38.885 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:17:39.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T15:17:39.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:17:39.886 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-04T15:17:39.886 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:17:39.955 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-03-04T15:17:39.985 DEBUG:teuthology.parallel:result is None 2024-03-04T15:17:42.912 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-03-04T15:17:57.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T15:17:57.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:17:59.862 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-04T15:18:06.898 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T15:18:06.898 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:18:06.898 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-03-04T15:18:06.912 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:18:06.912 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:18:06.912 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:18:06.912 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:18:07.913 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-04T15:18:07.913 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:18:07.978 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-03-04T15:18:08.003 DEBUG:teuthology.parallel:result is None 2024-03-04T15:18:15.248 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T15:18:30.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-03-04T15:18:30.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:18:30.608 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-03-04T15:18:30.971 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-03-04T15:18:30.971 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:18:30.971 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-03-04T15:18:30.986 INFO:teuthology.orchestra.run.smithi160.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:18:30.987 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:18:30.987 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:18:30.987 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:18:31.988 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-04T15:18:31.988 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:18:32.078 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2024-03-04T15:18:32.108 DEBUG:teuthology.parallel:result is None 2024-03-04T15:18:32.109 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T15:18:32.115 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T15:18:32.115 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:18:32.118 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:18:32.121 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:18:32.139 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T15:18:32.145 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T15:18:32.154 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T15:18:32.154 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:18:32.176 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:18:32.181 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:18:32.212 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T15:18:32.219 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T15:18:32.219 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:18:32.240 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:18:32.246 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:18:32.275 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:18:32.276 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:18:32.282 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:18:32.289 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:18:32.291 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:18:32.296 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:18:32.297 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T15:18:32.304 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T15:18:32.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:18:32.334 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:18:32.336 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:18:32.367 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T15:18:32.376 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T15:18:32.376 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:18:32.404 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:18:32.406 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:18:32.424 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:18:32.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:18:32.546 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:18:32.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:18:32.671 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:18:32.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:18:32.737 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:18:32.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:18:32.836 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:18:32.923 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:18:32.965 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:18:32.965 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:18:33.034 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:18:33.080 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:18:33.140 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:18:33.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:18:33.228 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:18:33.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:18:33.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-03-04T15:18:33.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-03-04T15:18:33.302 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-03-04T15:18:33.350 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:18:33.363 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:18:33.402 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:18:33.812 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T15:18:33.819 INFO:teuthology.task.internal:Starting timer... 2024-03-04T15:18:33.819 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T15:18:33.829 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T15:18:33.839 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-04T15:18:33.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-03-04T15:18:33.896 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-03-04T15:18:33.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-03-04T15:18:33.951 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-03-04T15:18:33.953 DEBUG:teuthology.orchestra.run.smithi160:> sudo service auditd rotate 2024-03-04T15:18:34.015 INFO:teuthology.orchestra.run.smithi160.stdout:Rotating logs: 2024-03-04T15:18:34.017 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T15:18:34.017 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-03-04T15:18:34.039 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-03-04T15:18:34.040 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-03-04T15:18:34.064 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-03-04T15:18:34.065 DEBUG:teuthology.orchestra.run.smithi160:> /usr/sbin/getenforce 2024-03-04T15:18:34.088 INFO:teuthology.orchestra.run.smithi160.stdout:Permissive 2024-03-04T15:18:34.089 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi160.front.sepia.ceph.com': 'permissive'} 2024-03-04T15:18:34.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:18:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:18:34.117 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:18:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:18:34.146 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:18:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:18:34.175 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T15:18:34.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:18:34.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:18:34.241 DEBUG:teuthology.orchestra.run.smithi160:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:18:34.274 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T15:18:34.283 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-04T15:18:34.288 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-04T15:18:34.428 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T15:18:34.444 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T15:18:34.445 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi064.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T15:23:07.840 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2024-03-04T15:23:07.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-04T15:23:07.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:23:07.978 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-04T15:23:08.068 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-04T15:23:08.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T15:23:08.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:23:08.267 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-03-04T15:23:08.349 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-03-04T15:23:08.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-03-04T15:23:08.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:23:08.520 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-03-04T15:23:08.599 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-03-04T15:23:08.599 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T15:23:08.612 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T15:23:08.612 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:23:08.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:23:08.616 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:23:08.616 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:23:08.618 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:23:08.619 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:23:08.650 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:23:08.667 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:23:08.669 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:23:08.670 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:23:08.687 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:23:08.703 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-03-04T15:23:08.703 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-03-04T15:23:08.712 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-03-04T15:23:08.717 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-03-04T15:23:08.720 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: ntpd: command not found 2024-03-04T15:23:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:23:08.731 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:23:08.735 INFO:teuthology.orchestra.run.smithi160.stdout:506 Cannot talk to daemon 2024-03-04T15:23:08.745 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:23:08.748 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:23:08.750 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:23:08.771 INFO:teuthology.orchestra.run.smithi160.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:23:08.803 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:23:08.804 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:23:08.805 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:23:08.805 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-04T15:23:08.805 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.805 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.806 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.834 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:23:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:23:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-03-04T15:23:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.837 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:23:08.838 INFO:teuthology.run_tasks:Running task pexec... 2024-03-04T15:23:08.850 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-04T15:23:08.850 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi053.front.sepia.ceph.com 2024-03-04T15:23:08.850 DEBUG:teuthology.orchestra.run.smithi053:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-04T15:23:08.851 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T15:23:08.851 DEBUG:teuthology.orchestra.run.smithi064:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-04T15:23:08.851 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi160.front.sepia.ceph.com 2024-03-04T15:23:08.852 DEBUG:teuthology.orchestra.run.smithi160:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-04T15:23:09.444 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:01 ago on Mon 04 Mar 2024 03:22:08 PM UTC. 2024-03-04T15:23:09.447 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:02 ago on Mon 04 Mar 2024 03:22:07 PM UTC. 2024-03-04T15:23:09.484 INFO:teuthology.orchestra.run.smithi160.stdout:Last metadata expiration check: 0:01:04 ago on Mon 04 Mar 2024 03:22:05 PM UTC. 2024-03-04T15:23:09.612 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-03-04T15:23:09.614 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T15:23:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2024-03-04T15:23:09.614 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T15:23:09.614 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-03-04T15:23:09.614 INFO:teuthology.orchestra.run.smithi064.stdout: runc x86_64 4:1.1.12-2.el9 CentOS-AppStream 3.1 M 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout:Install 1 Package 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 3.1 M 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 9.8 M 2024-03-04T15:23:09.615 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-03-04T15:23:09.617 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-03-04T15:23:09.618 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-04T15:23:09.618 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-03-04T15:23:09.618 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-04T15:23:09.618 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-03-04T15:23:09.618 INFO:teuthology.orchestra.run.smithi053.stdout: runc x86_64 4:1.1.12-2.el9 CentOS-AppStream 3.1 M 2024-03-04T15:23:09.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:23:09.619 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-03-04T15:23:09.619 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-03-04T15:23:09.619 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-03-04T15:23:09.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:23:09.619 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.1 M 2024-03-04T15:23:09.620 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 9.8 M 2024-03-04T15:23:09.620 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-03-04T15:23:09.649 INFO:teuthology.orchestra.run.smithi160.stdout:Dependencies resolved. 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: Package Architecture Version Repository Size 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout:Installing: 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: runc x86_64 4:1.1.12-2.el9 CentOS-AppStream 3.1 M 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction Summary 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout:================================================================================ 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout:Install 1 Package 2024-03-04T15:23:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:23:09.657 INFO:teuthology.orchestra.run.smithi160.stdout:Total download size: 3.1 M 2024-03-04T15:23:09.657 INFO:teuthology.orchestra.run.smithi160.stdout:Installed size: 9.8 M 2024-03-04T15:23:09.657 INFO:teuthology.orchestra.run.smithi160.stdout:Downloading Packages: 2024-03-04T15:23:09.895 INFO:teuthology.orchestra.run.smithi064.stdout:runc-1.1.12-2.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-03-04T15:23:09.896 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:23:09.896 INFO:teuthology.orchestra.run.smithi064.stdout:Total 11 MB/s | 3.1 MB 00:00 2024-03-04T15:23:09.896 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-03-04T15:23:09.901 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-03-04T15:23:09.902 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-03-04T15:23:09.952 INFO:teuthology.orchestra.run.smithi053.stdout:runc-1.1.12-2.el9.x86_64.rpm 9.6 MB/s | 3.1 MB 00:00 2024-03-04T15:23:09.952 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:23:09.952 INFO:teuthology.orchestra.run.smithi053.stdout:Total 9.5 MB/s | 3.1 MB 00:00 2024-03-04T15:23:09.952 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-03-04T15:23:09.952 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-03-04T15:23:09.953 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-03-04T15:23:09.956 INFO:teuthology.orchestra.run.smithi160.stdout:runc-1.1.12-2.el9.x86_64.rpm 10 MB/s | 3.1 MB 00:00 2024-03-04T15:23:09.960 INFO:teuthology.orchestra.run.smithi160.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:23:09.960 INFO:teuthology.orchestra.run.smithi160.stdout:Total 10 MB/s | 3.1 MB 00:00 2024-03-04T15:23:09.960 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction check 2024-03-04T15:23:09.962 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction check succeeded. 2024-03-04T15:23:09.962 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction test 2024-03-04T15:23:09.977 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-03-04T15:23:09.977 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-03-04T15:23:10.027 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-03-04T15:23:10.027 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-03-04T15:23:10.035 INFO:teuthology.orchestra.run.smithi160.stdout:Transaction test succeeded. 2024-03-04T15:23:10.035 INFO:teuthology.orchestra.run.smithi160.stdout:Running transaction 2024-03-04T15:23:10.186 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-03-04T15:23:10.238 INFO:teuthology.orchestra.run.smithi160.stdout: Preparing : 1/1 2024-03-04T15:23:10.252 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-03-04T15:23:10.293 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:10.333 INFO:teuthology.orchestra.run.smithi160.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:10.336 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:10.901 INFO:teuthology.orchestra.run.smithi160.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:11.118 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:11.118 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:23:11.118 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-03-04T15:23:11.119 INFO:teuthology.orchestra.run.smithi053.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-04T15:23:11.119 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:23:11.119 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-03-04T15:23:11.226 INFO:teuthology.orchestra.run.smithi160.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:11.226 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:23:11.226 INFO:teuthology.orchestra.run.smithi160.stdout:Installed: 2024-03-04T15:23:11.226 INFO:teuthology.orchestra.run.smithi160.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-04T15:23:11.226 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:23:11.226 INFO:teuthology.orchestra.run.smithi160.stdout:Complete! 2024-03-04T15:23:11.372 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-04T15:23:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:23:11.373 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-03-04T15:23:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-04T15:23:11.373 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-03-04T15:23:11.373 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-03-04T15:23:11.398 DEBUG:teuthology.parallel:result is None 2024-03-04T15:23:11.452 DEBUG:teuthology.parallel:result is None 2024-03-04T15:23:11.609 DEBUG:teuthology.parallel:result is None 2024-03-04T15:23:11.610 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-04T15:23:11.642 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-04T15:23:11.643 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:23:11.643 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:23:11.660 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:23:11.660 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-03-04T15:23:11.718 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:23:11.718 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:11.718 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-04T15:23:11.719 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:11.719 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:11.719 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:23:06.304955920 +0000 2024-03-04T15:23:11.719 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:22:35.588539664 +0000 2024-03-04T15:23:11.719 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:22:35.588539664 +0000 2024-03-04T15:23:11.719 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:22:35.588539664 +0000 2024-03-04T15:23:11.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:23:11.787 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:23:11.787 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:23:11.787 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000102131 s, 5.0 MB/s 2024-03-04T15:23:11.791 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:23:11.846 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-03-04T15:23:11.903 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:23:06.304955920 +0000 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:22:35.990532022 +0000 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:22:35.990532022 +0000 2024-03-04T15:23:11.904 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:22:35.990532022 +0000 2024-03-04T15:23:11.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:23:11.969 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:23:11.969 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:23:11.970 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.5306e-05 s, 5.4 MB/s 2024-03-04T15:23:11.971 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:23:12.026 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-03-04T15:23:12.080 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:23:12.080 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:12.080 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-04T15:23:12.080 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:12.080 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:12.080 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:23:06.304955920 +0000 2024-03-04T15:23:12.081 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:22:36.331525540 +0000 2024-03-04T15:23:12.081 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:22:36.331525540 +0000 2024-03-04T15:23:12.081 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:22:36.331525540 +0000 2024-03-04T15:23:12.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:23:12.147 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:23:12.147 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:23:12.147 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 7.8701e-05 s, 6.5 MB/s 2024-03-04T15:23:12.147 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:23:12.206 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:23:06.304955920 +0000 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:22:36.711518316 +0000 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:22:36.711518316 +0000 2024-03-04T15:23:12.261 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:22:36.711518316 +0000 2024-03-04T15:23:12.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:23:12.326 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:23:12.326 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:23:12.327 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.6236e-05 s, 5.3 MB/s 2024-03-04T15:23:12.327 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:23:12.382 DEBUG:teuthology.orchestra.run.smithi053:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-04T15:23:12.522 INFO:teuthology.orchestra.run.smithi053.stdout:loop 2024-03-04T15:23:12.524 INFO:tasks.nvme_loop:Connecting nvme_loop smithi053:/dev/vg_nvme/lv_1... 2024-03-04T15:23:12.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-04T15:23:12.561 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-04T15:23:12.598 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/vg_nvme/lv_11 2024-03-04T15:23:12.627 INFO:tasks.nvme_loop:Connecting nvme_loop smithi053:/dev/vg_nvme/lv_2... 2024-03-04T15:23:12.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-04T15:23:12.716 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-04T15:23:12.751 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/vg_nvme/lv_21 2024-03-04T15:23:12.779 INFO:tasks.nvme_loop:Connecting nvme_loop smithi053:/dev/vg_nvme/lv_3... 2024-03-04T15:23:12.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-04T15:23:12.858 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-04T15:23:12.897 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/vg_nvme/lv_31 2024-03-04T15:23:12.923 INFO:tasks.nvme_loop:Connecting nvme_loop smithi053:/dev/vg_nvme/lv_4... 2024-03-04T15:23:12.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-04T15:23:13.005 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-04T15:23:13.039 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/vg_nvme/lv_41 2024-03-04T15:23:13.066 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:23:13.067 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:23:13.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo nvme list 2024-03-04T15:23:13.187 INFO:teuthology.orchestra.run.smithi053.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-04T15:23:13.187 INFO:teuthology.orchestra.run.smithi053.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-04T15:23:13.187 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/nvme0n1 PHMB751300G8480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-04T15:23:13.187 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/nvme1n1 209ec5e6fafa3475bf02 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:13.187 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/nvme2n1 8a642221cab32acf0129 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:13.187 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/nvme3n1 9ad0515a017afc1b2914 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:13.187 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/nvme4n1 aec47f32dfe2106994f3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:13.188 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:23:13.188 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:23:13.188 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/scratch_devs 2024-03-04T15:23:13.257 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:23:13.257 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:23:13.276 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:23:13.276 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:23:06.339898984 +0000 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:22:38.089454650 +0000 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:22:38.089454650 +0000 2024-03-04T15:23:13.338 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:22:38.089454650 +0000 2024-03-04T15:23:13.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:23:13.410 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:23:13.410 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:23:13.410 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000196364 s, 2.6 MB/s 2024-03-04T15:23:13.411 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:23:13.471 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-03-04T15:23:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:23:13.529 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:13.529 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-04T15:23:13.529 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:13.529 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:13.530 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:23:06.339898984 +0000 2024-03-04T15:23:13.530 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:22:38.476447038 +0000 2024-03-04T15:23:13.530 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:22:38.476447038 +0000 2024-03-04T15:23:13.530 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:22:38.476447038 +0000 2024-03-04T15:23:13.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:23:13.603 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:23:13.603 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:23:13.603 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000194167 s, 2.6 MB/s 2024-03-04T15:23:13.603 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:23:13.661 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:23:06.339898984 +0000 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:22:38.871439269 +0000 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:22:38.871439269 +0000 2024-03-04T15:23:13.721 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:22:38.871439269 +0000 2024-03-04T15:23:13.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:23:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:23:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:23:13.789 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000209189 s, 2.4 MB/s 2024-03-04T15:23:13.790 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:23:13.847 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:23:06.339898984 +0000 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:22:39.270431421 +0000 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:22:39.270431421 +0000 2024-03-04T15:23:13.905 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:22:39.270431421 +0000 2024-03-04T15:23:13.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:23:13.971 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:23:13.971 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:23:13.971 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000174283 s, 2.9 MB/s 2024-03-04T15:23:13.972 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:23:14.028 DEBUG:teuthology.orchestra.run.smithi064:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-04T15:23:14.167 INFO:teuthology.orchestra.run.smithi064.stdout:loop 2024-03-04T15:23:14.168 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_1... 2024-03-04T15:23:14.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-04T15:23:14.207 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-03-04T15:23:14.248 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_11 2024-03-04T15:23:14.274 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_2... 2024-03-04T15:23:14.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-04T15:23:14.355 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-03-04T15:23:14.392 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_21 2024-03-04T15:23:14.433 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_3... 2024-03-04T15:23:14.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-04T15:23:14.511 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-03-04T15:23:14.548 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_31 2024-03-04T15:23:14.576 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_4... 2024-03-04T15:23:14.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-04T15:23:14.656 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-03-04T15:23:14.693 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_41 2024-03-04T15:23:14.720 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:23:14.720 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:23:14.778 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme list 2024-03-04T15:23:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-04T15:23:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-04T15:23:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme0n1 CVFT534400C2400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-04T15:23:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme1n1 40bcd40eb0a93e5d4bdd Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme2n1 6e083438a21556ffed39 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme3n1 6001aadf45c8b55e04a6 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:14.844 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme4n1 85e63a40c15460d9b55a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:14.845 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:23:14.846 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:23:14.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/scratch_devs 2024-03-04T15:23:14.912 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:23:14.913 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:23:14.932 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:23:14.933 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-03-04T15:23:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:23:14.990 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:14.991 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-04T15:23:14.991 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:14.991 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:14.991 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:23:06.324095777 +0000 2024-03-04T15:23:14.991 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:22:34.586529552 +0000 2024-03-04T15:23:14.991 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:22:34.586529552 +0000 2024-03-04T15:23:14.991 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:22:34.586529552 +0000 2024-03-04T15:23:14.991 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:23:15.065 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:23:15.065 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:23:15.065 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000220756 s, 2.3 MB/s 2024-03-04T15:23:15.066 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:23:15.129 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-03-04T15:23:15.186 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:23:15.186 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:15.187 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-04T15:23:15.187 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:15.187 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:15.187 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:23:06.324095777 +0000 2024-03-04T15:23:15.187 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:22:34.979524181 +0000 2024-03-04T15:23:15.187 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:22:34.979524181 +0000 2024-03-04T15:23:15.187 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:22:34.979524181 +0000 2024-03-04T15:23:15.187 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:23:15.263 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:23:15.263 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:23:15.263 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000232254 s, 2.2 MB/s 2024-03-04T15:23:15.264 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:23:15.321 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:23:06.324095777 +0000 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:22:35.392518536 +0000 2024-03-04T15:23:15.377 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:22:35.392518536 +0000 2024-03-04T15:23:15.378 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:22:35.392518536 +0000 2024-03-04T15:23:15.378 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:23:15.448 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:23:15.449 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:23:15.449 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000183953 s, 2.8 MB/s 2024-03-04T15:23:15.450 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:23:15.506 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:23:06.324095777 +0000 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:22:35.774513315 +0000 2024-03-04T15:23:15.565 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:22:35.774513315 +0000 2024-03-04T15:23:15.566 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:22:35.774513315 +0000 2024-03-04T15:23:15.566 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:23:15.637 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:23:15.637 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:23:15.637 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000154898 s, 3.3 MB/s 2024-03-04T15:23:15.637 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:23:15.693 DEBUG:teuthology.orchestra.run.smithi160:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-04T15:23:15.826 INFO:teuthology.orchestra.run.smithi160.stdout:loop 2024-03-04T15:23:15.827 INFO:tasks.nvme_loop:Connecting nvme_loop smithi160:/dev/vg_nvme/lv_1... 2024-03-04T15:23:15.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-04T15:23:15.866 INFO:teuthology.orchestra.run.smithi160.stdout:1 2024-03-04T15:23:15.903 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_11 2024-03-04T15:23:15.944 INFO:tasks.nvme_loop:Connecting nvme_loop smithi160:/dev/vg_nvme/lv_2... 2024-03-04T15:23:15.944 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-04T15:23:16.024 INFO:teuthology.orchestra.run.smithi160.stdout:1 2024-03-04T15:23:16.063 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_21 2024-03-04T15:23:16.092 INFO:tasks.nvme_loop:Connecting nvme_loop smithi160:/dev/vg_nvme/lv_3... 2024-03-04T15:23:16.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-04T15:23:16.171 INFO:teuthology.orchestra.run.smithi160.stdout:1 2024-03-04T15:23:16.208 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_31 2024-03-04T15:23:16.240 INFO:tasks.nvme_loop:Connecting nvme_loop smithi160:/dev/vg_nvme/lv_4... 2024-03-04T15:23:16.240 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-04T15:23:16.319 INFO:teuthology.orchestra.run.smithi160.stdout:1 2024-03-04T15:23:16.356 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/vg_nvme/lv_41 2024-03-04T15:23:16.391 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:23:16.392 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:23:16.447 DEBUG:teuthology.orchestra.run.smithi160:> sudo nvme list 2024-03-04T15:23:16.513 INFO:teuthology.orchestra.run.smithi160.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-04T15:23:16.513 INFO:teuthology.orchestra.run.smithi160.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-04T15:23:16.513 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/nvme0n1 CVFT623300JY400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-04T15:23:16.513 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/nvme1n1 baf30562f5ac9b412ad7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:16.513 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/nvme2n1 3facdfda0dbef714dd08 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:16.513 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/nvme3n1 f08755519e4a555b6d1d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:16.513 INFO:teuthology.orchestra.run.smithi160.stdout:/dev/nvme4n1 820b20529b6356bab516 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-04T15:23:16.514 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:23:16.514 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:23:16.514 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/scratch_devs 2024-03-04T15:23:16.617 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-04T15:23:16.835 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'} 2024-03-04T15:23:16.838 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:23:16.840 INFO:tasks.cephadm:Cluster fsid is 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:23:16.841 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-04T15:23:16.841 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.53', 'mon.b': '172.21.15.64', 'mon.c': '172.21.15.160'} 2024-03-04T15:23:16.841 INFO:tasks.cephadm:First mon is mon.a on smithi053 2024-03-04T15:23:16.841 INFO:tasks.cephadm:First mgr is y 2024-03-04T15:23:16.841 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-04T15:23:16.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-03-04T15:23:16.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-03-04T15:23:16.942 DEBUG:teuthology.orchestra.run.smithi160:> sudo hostname $(hostname -s) 2024-03-04T15:23:16.969 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-04T15:23:16.969 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:23:17.214 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77294/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1826-g9ca0b5de', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1826.g9ca0b5de'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-04 12:37:35.725599', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-asm0deuz-testing-2024-03-04-1211', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-04T15:23:17.385 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-asm0deuz-testing-2024-03-04-1211, sha1 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:23:17.387 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm 2024-03-04T15:23:17.388 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:17.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:23 /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:17.842 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:18.449 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:23 /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:18.449 DEBUG:teuthology.orchestra.run.smithi160:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:19.053 INFO:teuthology.orchestra.run.smithi160.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:23 /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:19.054 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:19.073 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:19.095 DEBUG:teuthology.orchestra.run.smithi160:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:23:19.137 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 on all hosts... 2024-03-04T15:23:19.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:23:19.140 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:23:19.144 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:23:19.983 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:23:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:23:20.075 INFO:teuthology.orchestra.run.smithi160.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:23:48.653 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-03-04T15:23:48.653 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:23:48.653 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:23:48.653 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-03-04T15:23:48.653 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:23:48.653 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-03-04T15:23:48.653 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-03-04T15:24:03.482 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-04T15:24:03.482 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:24:03.482 INFO:teuthology.orchestra.run.smithi053.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:24:03.482 INFO:teuthology.orchestra.run.smithi053.stdout: "repo_digests": [ 2024-03-04T15:24:03.482 INFO:teuthology.orchestra.run.smithi053.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:24:03.482 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-03-04T15:24:03.482 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-04T15:24:13.232 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-03-04T15:24:13.233 INFO:teuthology.orchestra.run.smithi160.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:24:13.233 INFO:teuthology.orchestra.run.smithi160.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:24:13.233 INFO:teuthology.orchestra.run.smithi160.stdout: "repo_digests": [ 2024-03-04T15:24:13.233 INFO:teuthology.orchestra.run.smithi160.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:24:13.233 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-03-04T15:24:13.233 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-03-04T15:24:13.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-03-04T15:24:13.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-03-04T15:24:13.329 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph 2024-03-04T15:24:13.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-03-04T15:24:13.388 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-03-04T15:24:13.420 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /etc/ceph 2024-03-04T15:24:13.448 INFO:tasks.cephadm:Writing seed config... 2024-03-04T15:24:13.449 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-04T15:24:13.449 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-04T15:24:13.449 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-04T15:24:13.450 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-04T15:24:13.450 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-04T15:24:13.450 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-04T15:24:13.450 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-04T15:24:13.450 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-04T15:24:13.451 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-04T15:24:13.451 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-04T15:24:13.453 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:24:13.453 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-04T15:24:13.472 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1fe97ada-da3b-11ee-95c6-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-04T15:24:13.472 DEBUG:teuthology.orchestra.run.smithi053:mon.a> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a.service 2024-03-04T15:24:13.517 DEBUG:teuthology.orchestra.run.smithi053:mgr.y> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.y.service 2024-03-04T15:24:13.559 INFO:tasks.cephadm:Bootstrapping... 2024-03-04T15:24:13.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -v bootstrap --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.53 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:24:13.757 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:24:13.758 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282', '-v', 'bootstrap', '--fsid', '1fe97ada-da3b-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.53', '--skip-admin-label'] 2024-03-04T15:24:13.784 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:24:13.784 INFO:teuthology.orchestra.run.smithi053.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-04T15:24:13.784 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying podman|docker is present... 2024-03-04T15:24:13.807 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:24:13.808 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying lvm2 is present... 2024-03-04T15:24:13.808 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying time synchronization is in place... 2024-03-04T15:24:13.815 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:24:13.815 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:24:13.820 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:24:13.820 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-03-04T15:24:13.827 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout enabled 2024-03-04T15:24:13.834 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout active 2024-03-04T15:24:13.834 INFO:teuthology.orchestra.run.smithi053.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:24:13.834 INFO:teuthology.orchestra.run.smithi053.stdout:Repeating the final host check... 2024-03-04T15:24:13.858 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:24:13.858 INFO:teuthology.orchestra.run.smithi053.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-04T15:24:13.858 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl is present 2024-03-04T15:24:13.858 INFO:teuthology.orchestra.run.smithi053.stdout:lvcreate is present 2024-03-04T15:24:13.865 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:24:13.865 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:24:13.873 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:24:13.873 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-03-04T15:24:13.878 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout enabled 2024-03-04T15:24:13.886 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout active 2024-03-04T15:24:13.886 INFO:teuthology.orchestra.run.smithi053.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:24:13.886 INFO:teuthology.orchestra.run.smithi053.stdout:Host looks OK 2024-03-04T15:24:13.886 INFO:teuthology.orchestra.run.smithi053.stdout:Cluster fsid: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:13.886 INFO:teuthology.orchestra.run.smithi053.stdout:Acquiring lock 140672136508800 on /run/cephadm/1fe97ada-da3b-11ee-95c6-87774f69a715.lock 2024-03-04T15:24:13.886 INFO:teuthology.orchestra.run.smithi053.stdout:Lock 140672136508800 acquired on /run/cephadm/1fe97ada-da3b-11ee-95c6-87774f69a715.lock 2024-03-04T15:24:13.886 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying IP 172.21.15.53 port 3300 ... 2024-03-04T15:24:13.887 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying IP 172.21.15.53 port 6789 ... 2024-03-04T15:24:13.887 INFO:teuthology.orchestra.run.smithi053.stdout:Base mon IP(s) is [172.21.15.53:3300, 172.21.15.53:6789], mon addrv is [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-03-04T15:24:13.889 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.53 metric 100 2024-03-04T15:24:13.889 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.53 metric 100 2024-03-04T15:24:13.892 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-04T15:24:13.892 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-04T15:24:13.892 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-04T15:24:13.894 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-04T15:24:13.895 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-04T15:24:13.895 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:24:13.895 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-04T15:24:13.895 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-03-04T15:24:13.895 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:24:13.896 INFO:teuthology.orchestra.run.smithi053.stdout:Mon IP `172.21.15.53` is in CIDR network `172.21.0.0/20` 2024-03-04T15:24:13.896 INFO:teuthology.orchestra.run.smithi053.stdout:Mon IP `172.21.15.53` is in CIDR network `172.21.0.0/20` 2024-03-04T15:24:13.896 INFO:teuthology.orchestra.run.smithi053.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-04T15:24:13.896 INFO:teuthology.orchestra.run.smithi053.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-04T15:24:13.896 INFO:teuthology.orchestra.run.smithi053.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:24:14.353 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stdout 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:24:14.354 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:24:14.354 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Getting image source signatures 2024-03-04T15:24:14.354 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-04T15:24:14.354 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying blob sha256:64fd3420507ff7dcf06d181c53a3dd949a37f619fe84875331c583b3b5f6f668 2024-03-04T15:24:14.354 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Copying config sha256:0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:24:14.354 INFO:teuthology.orchestra.run.smithi053.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-04T15:24:18.960 INFO:teuthology.orchestra.run.smithi053.stdout:ceph: stdout ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T15:24:18.960 INFO:teuthology.orchestra.run.smithi053.stdout:Ceph version: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T15:24:18.960 INFO:teuthology.orchestra.run.smithi053.stdout:Extracting ceph user uid/gid from container image... 2024-03-04T15:24:19.768 INFO:teuthology.orchestra.run.smithi053.stdout:stat: stdout 167 167 2024-03-04T15:24:19.768 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial keys... 2024-03-04T15:24:20.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQCk5+Vl0aLfBhAAHo5XnvEcRD1LIJlW2emRLw== 2024-03-04T15:24:21.574 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQCk5+Vl/NJQOhAA+Jhx4sGclak6B6jXS7O84A== 2024-03-04T15:24:22.520 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph-authtool: stdout AQCl5+Vl3aEiOBAA5dzjn24cZahmJhOyiiNd8A== 2024-03-04T15:24:22.521 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial monmap... 2024-03-04T15:24:23.643 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:24:23.643 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool for a [v2:172.21.15.53:3300,v1:172.21.15.53:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:setting min_mon_release = quincy 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: set fsid to 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:24:23.644 INFO:teuthology.orchestra.run.smithi053.stdout:Creating mon... 2024-03-04T15:24:24.798 INFO:teuthology.orchestra.run.smithi053.stdout:create mon.a on 2024-03-04T15:24:25.154 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-04T15:24:25.334 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715.target. 2024-03-04T15:24:25.334 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715.target. 2024-03-04T15:24:25.564 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a 2024-03-04T15:24:25.564 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-04T15:24:25.722 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715.target.wants/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@.service. 2024-03-04T15:24:26.124 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:25 smithi053 podman[25209]: 2024-03-04 15:24:25.870189313 +0000 UTC m=+0.018587497 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:24:26.124 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:26 smithi053 podman[25209]: 2024-03-04 15:24:26.12278815 +0000 UTC m=+0.271186342 container init 7314ea5be099a30156cd49725fd75876ee56ded5e766078e97586758b567c6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-04T15:24:26.206 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:24:26.206 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-03-04T15:24:26.213 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:24:26.213 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-03-04T15:24:26.213 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-03-04T15:24:26.213 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:24:26.213 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mon to start... 2024-03-04T15:24:26.214 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mon... 2024-03-04T15:24:26.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:26 smithi053 podman[25209]: 2024-03-04 15:24:26.128563477 +0000 UTC m=+0.276961661 container start 7314ea5be099a30156cd49725fd75876ee56ded5e766078e97586758b567c6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:24:26.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:26 smithi053 bash[25209]: 7314ea5be099a30156cd49725fd75876ee56ded5e766078e97586758b567c6de 2024-03-04T15:24:26.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:26 smithi053 systemd[1]: Started Ceph mon.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:24:26.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:26 smithi053 ceph-mon[25246]: mkfs 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:26.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:26 smithi053 ceph-mon[25246]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:24:27.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:27 smithi053 ceph-mon[25246]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:24:27.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:27 smithi053 ceph-mon[25246]: monmap e1: 1 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:24:27.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:27 smithi053 ceph-mon[25246]: fsmap 2024-03-04T15:24:27.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:27 smithi053 ceph-mon[25246]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:24:27.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:27 smithi053 ceph-mon[25246]: mgrmap e1: no daemons active 2024-03-04T15:24:27.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:27 smithi053 ceph-mon[25246]: from='client.? 172.21.15.53:0/960087882' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-04T15:24:27.392 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout cluster: 2024-03-04T15:24:27.392 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout id: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout services: 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.544996s) 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout data: 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout pgs: 2024-03-04T15:24:27.393 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:27.394 INFO:teuthology.orchestra.run.smithi053.stdout:mon is available 2024-03-04T15:24:27.394 INFO:teuthology.orchestra.run.smithi053.stdout:Assimilating anything we can from ceph.conf... 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout fsid = 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.53:3300,v1:172.21.15.53:6789] 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:28.373 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:24:28.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-04T15:24:28.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:24:28.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:28.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:24:28.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:24:28.374 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:24:28.374 INFO:teuthology.orchestra.run.smithi053.stdout:Generating new minimal ceph.conf... 2024-03-04T15:24:28.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:28 smithi053 ceph-mon[25246]: from='client.? 172.21.15.53:0/3817663401' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:24:28.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:28 smithi053 ceph-mon[25246]: from='client.? 172.21.15.53:0/3817663401' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T15:24:29.452 INFO:teuthology.orchestra.run.smithi053.stdout:Restarting the monitor... 2024-03-04T15:24:29.466 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:29 smithi053 ceph-mon[25246]: from='client.? 172.21.15.53:0/2718031259' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:24:29.831 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:29 smithi053 systemd[1]: Stopping Ceph mon.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:24:29.831 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:29 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a[25223]: 2024-03-04T15:24:29.712+0000 7f2461058700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:24:29.831 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:29 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a[25223]: 2024-03-04T15:24:29.712+0000 7f2461058700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T15:24:30.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:29 smithi053 podman[25576]: 2024-03-04 15:24:29.926512636 +0000 UTC m=+0.299490448 container died 7314ea5be099a30156cd49725fd75876ee56ded5e766078e97586758b567c6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T15:24:30.453 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 podman[25576]: 2024-03-04 15:24:30.250699639 +0000 UTC m=+0.623677433 container cleanup 7314ea5be099a30156cd49725fd75876ee56ded5e766078e97586758b567c6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T15:24:30.453 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 bash[25576]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a 2024-03-04T15:24:30.862 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 podman[25594]: 2024-03-04 15:24:30.451726651 +0000 UTC m=+0.522810699 container remove 7314ea5be099a30156cd49725fd75876ee56ded5e766078e97586758b567c6de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T15:24:30.862 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-04T15:24:30.862 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 systemd[1]: Stopped Ceph mon.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:24:30.862 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 systemd[1]: Starting Ceph mon.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:24:31.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 podman[25700]: 2024-03-04 15:24:30.860822523 +0000 UTC m=+0.095336478 container create 4625c153d19876b7caa8b7557dbfbb8b5fc7ae1b5b67627e8d79ed9b4f4e9333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:30 smithi053 podman[25700]: 2024-03-04 15:24:30.783148226 +0000 UTC m=+0.017662147 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 podman[25700]: 2024-03-04 15:24:31.071336874 +0000 UTC m=+0.305850788 container init 4625c153d19876b7caa8b7557dbfbb8b5fc7ae1b5b67627e8d79ed9b4f4e9333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 podman[25700]: 2024-03-04 15:24:31.077153621 +0000 UTC m=+0.311667534 container start 4625c153d19876b7caa8b7557dbfbb8b5fc7ae1b5b67627e8d79ed9b4f4e9333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 7 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: pidfile_write: ignore empty --pid-file 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: load: jerasure load: lrc 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Git sha 0 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:24:31.137 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: DB SUMMARY 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: DB Session ID: U2BYKDKSKYLPJ0RLFCXZ 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84625 ; 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.env: 0x55effc47db20 2024-03-04T15:24:31.138 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.info_log: 0x55effe10f8e0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.statistics: (nil) 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.use_fsync: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:24:31.139 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.db_log_dir: 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.wal_dir: 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.write_buffer_manager: 0x55effe0e3400 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:24:31.140 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.unordered_write: 0 2024-03-04T15:24:31.141 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:24:31.142 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:24:31.142 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:24:31.142 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:24:31.142 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.row_cache: None 2024-03-04T15:24:31.142 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.wal_filter: None 2024-03-04T15:24:31.142 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:24:31.142 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.wal_compression: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:24:31.143 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_open_files: -1 2024-03-04T15:24:31.144 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:24:31.145 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:24:31.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:24:31.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:24:31.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:24:31.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Compression algorithms supported: 2024-03-04T15:24:31.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:24:31.146 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kZSTD supported: 0 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.merge_operator: 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_filter: None 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:24:31.147 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55effe10f9e0) 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_type: 0 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: data_block_index_type: 0 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_shortening: 1 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: checksum: 4 2024-03-04T15:24:31.148 INFO:journalctl@ceph.mon.a.smithi053.stdout: no_block_cache: 0 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache: 0x55effe115090 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_options: 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: capacity : 536870912 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: num_shard_bits : 4 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: strict_capacity_limit : 0 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_cache_compressed: (nil) 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: persistent_cache: (nil) 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_size: 4096 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_size_deviation: 10 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_restart_interval: 16 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: index_block_restart_interval: 1 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: metadata_block_size: 4096 2024-03-04T15:24:31.149 INFO:journalctl@ceph.mon.a.smithi053.stdout: partition_filters: 0 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: use_delta_encoding: 1 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: filter_policy: bloomfilter 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: whole_key_filtering: 1 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: verify_compression: 0 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: format_version: 5 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: enable_index_compression: 1 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: block_align: 0 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: max_auto_readahead_size: 262144 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: prepopulate_block_cache: 0 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:24:31.150 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression: NoCompression 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.num_levels: 7 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:24:31.151 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:24:31.152 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:24:31.153 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:24:31.154 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:24:31.155 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.ttl: 2592000 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:24:31.156 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8cb6538d-d965-4b0d-b6a3-6fc74152be96 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565871105614, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565871106460, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81250, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79437, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73746, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709565871, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8cb6538d-d965-4b0d-b6a3-6fc74152be96", "db_session_id": "U2BYKDKSKYLPJ0RLFCXZ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565871106543, "job": 1, "event": "recovery_finished"} 2024-03-04T15:24:31.157 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55effe208000 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: DB pointer 0x55effe1f2000 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** DB Stats ** 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:24:31.158 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** Compaction Stats [default] ** 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: L0 2/0 81.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: Sum 2/0 81.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** Compaction Stats [default] ** 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 145.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:24:31.159 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: Cumulative compaction: 0.00 GB write, 26.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: Interval compaction: 0.00 GB write, 26.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: Block cache BinnedLRUCache@0x55effe115090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: starting mon.a rank 0 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:31.160 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???) e1 preinit fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???).mds e1 new map 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???).mds e1 print_map 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout: e1 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout: legacy client fscid: -1 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout: 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout: No filesystems configured 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:24:31.161 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:24:31.162 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-04T15:24:31.162 INFO:teuthology.orchestra.run.smithi053.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-04T15:24:31.411 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 bash[25700]: 4625c153d19876b7caa8b7557dbfbb8b5fc7ae1b5b67627e8d79ed9b4f4e9333 2024-03-04T15:24:31.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 systemd[1]: Started Ceph mon.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:24:31.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:24:31.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: monmap e1: 1 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:24:31.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: fsmap 2024-03-04T15:24:31.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:24:31.412 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:31 smithi053 ceph-mon[25738]: mgrmap e1: no daemons active 2024-03-04T15:24:32.298 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-04T15:24:32.319 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:24:32.319 INFO:teuthology.orchestra.run.smithi053.stdout:Creating mgr... 2024-03-04T15:24:32.320 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-04T15:24:32.320 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-04T15:24:32.570 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.y 2024-03-04T15:24:32.570 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.y.service: Unit ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.y.service not loaded. 2024-03-04T15:24:32.722 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715.target.wants/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@.service. 2024-03-04T15:24:33.074 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:32 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/2006522219' entity='client.admin' 2024-03-04T15:24:33.074 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:32 smithi053 podman[25979]: 2024-03-04 15:24:32.858577671 +0000 UTC m=+0.017340299 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:24:33.297 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:24:33.298 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-03-04T15:24:33.305 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:24:33.305 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-03-04T15:24:33.305 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-03-04T15:24:33.305 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:24:33.314 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:24:33.314 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-03-04T15:24:33.322 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:24:33.322 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-03-04T15:24:33.322 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-03-04T15:24:33.322 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-04T15:24:33.323 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr to start... 2024-03-04T15:24:33.323 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr... 2024-03-04T15:24:33.386 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:33 smithi053 podman[25979]: 2024-03-04 15:24:33.172288247 +0000 UTC m=+0.331050860 container init 5724c6d25cb1eda7e865bf7f73919d14635df5a728a80d3ef92af280edced264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T15:24:33.386 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:33 smithi053 podman[25979]: 2024-03-04 15:24:33.178031879 +0000 UTC m=+0.336794492 container start 5724c6d25cb1eda7e865bf7f73919d14635df5a728a80d3ef92af280edced264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:24:33.386 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:33 smithi053 bash[25979]: 5724c6d25cb1eda7e865bf7f73919d14635df5a728a80d3ef92af280edced264 2024-03-04T15:24:33.386 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:33 smithi053 systemd[1]: Started Ceph mgr.y for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:24:34.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:33 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/337609423' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:24:34.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:33 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:33.748+0000 7f452c773200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:24:34.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:33 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:33.846+0000 7f452c773200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:24:34.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:34.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:24:34.509 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "1fe97ada-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:34.510 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:24:34.512 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:34.513 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:24:34.513 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:24:34.513 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:24:34.513 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:24:34.513 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:24:34.513 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:24:34.517 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:24:34.518 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:24:26.197715+0000", 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:24:34.519 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (1/15)... 2024-03-04T15:24:34.757 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:34 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:34.517+0000 7f452c773200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:24:34.757 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:34 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:34.611+0000 7f452c773200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:24:35.135 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:34 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:34.753+0000 7f452c773200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:24:35.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:34 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:34.850+0000 7f452c773200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:24:35.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:34 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:34.934+0000 7f452c773200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:24:35.503 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:35 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:35.207+0000 7f452c773200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:24:35.503 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:35 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:35.318+0000 7f452c773200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:24:35.503 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:35 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:35.405+0000 7f452c773200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:24:35.503 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:35 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:35.501+0000 7f452c773200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:24:35.887 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:35 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:35.678+0000 7f452c773200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:24:36.885 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:36 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:36.675+0000 7f452c773200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:24:36.885 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:36 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:36.811+0000 7f452c773200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:24:37.248 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:37 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/802817463' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:24:37.248 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:37 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:37.154+0000 7f452c773200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:24:37.248 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:37 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:37.246+0000 7f452c773200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:24:37.593 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:37 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:37.445+0000 7f452c773200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:24:37.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:37.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:24:37.598 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "1fe97ada-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-04T15:24:37.599 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:24:37.600 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:37.606 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:37.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:24:37.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:37.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:24:26.197715+0000", 2024-03-04T15:24:37.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:37.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:37.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:24:37.608 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:24:37.608 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (2/15)... 2024-03-04T15:24:37.885 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:37 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:37.632+0000 7f452c773200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:24:38.182 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:37 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:37.918+0000 7f452c773200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:24:38.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:38 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:38.180+0000 7f452c773200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:24:38.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:38 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:38.419+0000 7f452c773200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:24:39.637 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:39 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:39.239+0000 7f452c773200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:24:39.979 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:39 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:39.851+0000 7f452c773200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:24:39.979 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:39 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:39.952+0000 7f452c773200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:24:40.252 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:40.207+0000 7f452c773200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1073625732' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: Activating manager daemon y 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: mgrmap e2: y(active, starting, since 0.00420693s) 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:24:40.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:40 smithi053 ceph-mon[25738]: Manager daemon y is now available 2024-03-04T15:24:40.653 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:40.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:24:40.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "1fe97ada-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:24:40.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:24:40.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:24:40.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:24:40.654 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:24:40.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:40.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:24:40.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:24:40.655 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:24:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:24:40.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:24:40.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:24:40.657 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:24:40.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:40.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:24:40.661 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:24:40.666 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:40.667 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:24:26.197715+0000", 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:24:40.668 INFO:teuthology.orchestra.run.smithi053.stdout:mgr not available, waiting (3/15)... 2024-03-04T15:24:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:41 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:24:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:41 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:24:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:41 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' 2024-03-04T15:24:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:41 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' 2024-03-04T15:24:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:41 smithi053 ceph-mon[25738]: from='mgr.14100 172.21.15.53:0/4142469839' entity='mgr.y' 2024-03-04T15:24:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:41 smithi053 ceph-mon[25738]: mgrmap e3: y(active, since 1.00769s) 2024-03-04T15:24:43.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:43 smithi053 ceph-mon[25738]: mgrmap e4: y(active, since 2s) 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsid": "1fe97ada-da3b-11ee-95c6-87774f69a715", 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:43.953 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:24:43.954 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:24:43.955 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:24:43.955 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:24:43.955 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:24:43.955 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:24:43.960 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:43.960 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:24:43.960 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:24:43.960 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:24:43.960 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:24:43.961 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:24:26.197715+0000", 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:24:43.962 INFO:teuthology.orchestra.run.smithi053.stdout:mgr is available 2024-03-04T15:24:44.306 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:44 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3463992252' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:24:45.337 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:45 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3601643345' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:24:45.337 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:45 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3601643345' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-04T15:24:45.343 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:45.344 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout fsid = 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:24:45.345 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:24:45.346 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:24:45.346 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling cephadm module... 2024-03-04T15:24:46.576 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:46 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/491746370' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-04T15:24:46.576 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:46 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ignoring --setuser ceph since I am not root 2024-03-04T15:24:46.576 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:46 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ignoring --setgroup ceph since I am not root 2024-03-04T15:24:47.106 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:46 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:46.831+0000 7fd92a5d3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:24:47.106 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:46 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:46.931+0000 7fd92a5d3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:24:47.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:47 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/491746370' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-04T15:24:47.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:47 smithi053 ceph-mon[25738]: mgrmap e5: y(active, since 6s) 2024-03-04T15:24:47.640 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:47 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:47.601+0000 7fd92a5d3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:24:47.943 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:47 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:47.699+0000 7fd92a5d3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:24:47.943 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:47 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:47.842+0000 7fd92a5d3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:24:47.943 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:47 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:47.941+0000 7fd92a5d3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:24:48.206 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:48.027+0000 7fd92a5d3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:24:48.209 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:24:48.210 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-04T15:24:48.210 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:24:48.210 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-04T15:24:48.210 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:24:48.210 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:24:48.210 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for the mgr to restart... 2024-03-04T15:24:48.210 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr epoch 5... 2024-03-04T15:24:48.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:48 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1536188807' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:24:48.464 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:48.297+0000 7fd92a5d3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:24:48.464 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:48.410+0000 7fd92a5d3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:24:48.783 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:48.502+0000 7fd92a5d3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:24:48.783 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:48.599+0000 7fd92a5d3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:24:49.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:48.780+0000 7fd92a5d3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:24:50.135 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:49 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:49.766+0000 7fd92a5d3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:24:50.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:49 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:49.900+0000 7fd92a5d3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:24:50.531 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:50 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:50.236+0000 7fd92a5d3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:24:50.531 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:50 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:50.328+0000 7fd92a5d3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:24:50.531 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:50 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:50.528+0000 7fd92a5d3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:24:50.885 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:50 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:50.716+0000 7fd92a5d3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:24:51.271 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:51 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:51.008+0000 7fd92a5d3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:24:51.271 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:51 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:51.269+0000 7fd92a5d3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:24:51.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:51 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:51.509+0000 7fd92a5d3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:24:52.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:52 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:52.335+0000 7fd92a5d3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:24:53.239 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:52 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:52.916+0000 7fd92a5d3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:24:53.239 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:53.016+0000 7fd92a5d3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:24:53.239 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:24:53.236+0000 7fd92a5d3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: Active manager daemon y restarted 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: Activating manager daemon y 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: osdmap e2: 0 total, 0 up, 0 in 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: mgrmap e6: y(active, starting, since 0.00326964s) 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:24:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:24:53.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:24:53.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: Manager daemon y is now available 2024-03-04T15:24:53.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:24:53.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:24:53.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:24:53.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:53 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:24:54.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:54 smithi053 ceph-mon[25738]: Found migration_current of "None". Setting to last migration. 2024-03-04T15:24:54.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:54 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:24:54.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:54 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:24:54.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:54 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:24:54.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:54 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:24:54.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:54 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:24:54.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:54 smithi053 ceph-mon[25738]: mgrmap e7: y(active, since 1.00495s) 2024-03-04T15:24:54.835 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:24:54.835 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-04T15:24:54.835 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:24:54.835 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:24:54.835 INFO:teuthology.orchestra.run.smithi053.stdout:mgr epoch 5 is available 2024-03-04T15:24:54.835 INFO:teuthology.orchestra.run.smithi053.stdout:Setting orchestrator backend to cephadm... 2024-03-04T15:24:55.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:55 smithi053 ceph-mon[25738]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:24:55.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:55 smithi053 ceph-mon[25738]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:24:56.529 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:56 smithi053 ceph-mon[25738]: [04/Mar/2024:15:24:55] ENGINE Bus STARTING 2024-03-04T15:24:56.529 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:56 smithi053 ceph-mon[25738]: mgrmap e8: y(active, since 2s) 2024-03-04T15:24:56.529 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:56 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:24:56.529 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:56 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:24:56.529 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:56 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:24:57.398 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:57 smithi053 ceph-mon[25738]: [04/Mar/2024:15:24:55] ENGINE Serving on http://172.21.15.53:8765 2024-03-04T15:24:57.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:57 smithi053 ceph-mon[25738]: [04/Mar/2024:15:24:55] ENGINE Serving on https://172.21.15.53:7150 2024-03-04T15:24:57.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:57 smithi053 ceph-mon[25738]: [04/Mar/2024:15:24:55] ENGINE Bus STARTED 2024-03-04T15:24:57.399 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:57 smithi053 ceph-mon[25738]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:24:57.401 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-04T15:24:57.401 INFO:teuthology.orchestra.run.smithi053.stdout:Generating ssh key... 2024-03-04T15:24:58.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-mon[25738]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:24:59.112 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: Generating public/private rsa key pair. 2024-03-04T15:24:59.112 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: Your identification has been saved in /tmp/tmp8ssnbffh/key. 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: Your public key has been saved in /tmp/tmp8ssnbffh/key.pub. 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: The key fingerprint is: 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: SHA256:1RvasB6BBe73N0va89PJxhGTSVTh4+sJSrjT/Gehdis ceph-1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: The key's randomart image is: 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: +---[RSA 3072]----+ 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | ... .o+| 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | . o . .. | 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | o + o .oo| 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | . . * o.=.| 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | S = o .o| 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | o.o o.| 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | .+...B.=| 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | .oo.E+%o| 2024-03-04T15:24:59.113 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: | ...+.O*+| 2024-03-04T15:24:59.114 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:24:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: +----[SHA256]-----+ 2024-03-04T15:24:59.363 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:59 smithi053 ceph-mon[25738]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:24:59.363 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:59 smithi053 ceph-mon[25738]: Generating ssh key... 2024-03-04T15:24:59.363 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:59 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:24:59.363 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:24:59 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:25:00.626 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:00 smithi053 ceph-mon[25738]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:00.630 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6w2F8shCirH3X7cTPoWWHh0mC43mXuq+GFHjPhp1rrhOC92F6vnyCEb6dC3Gp4AthuARd3ftYIeMtKoIaHJVQyXxc5MMJIzoo6jHfgFKMc2zVJD1MbloKVUplJcMjzJn/0/MTG2AG9ynq4zx2LMEb2IQ4OHjHhBfG0Dv2/3voEhkqBB2B1B/jQm01BfOWYm/NpZ60mSIjmnfddKHeiqtCVVlXZGfOXoacm4hpXo2pQhb21b32te74SHCt7cB2IsQdJL6fAsAxGHAux/OwyDOe4nAIxik4GeilfO7xvwwosAa2YaRXNv5vp3jZjMMSCyEe5cHNk5pKyoIuMgqGYjrCHNRz+5Qcu5Q08eOVGVTKZ7L5MR3Rl9FvNe/VIB65oDHSf9B6RY/i8CL//hmj/rcp2UnGl1lyRLSj4YmPwSqYjS8AFq1g+YllaQf9jwU40gq92NioCWQEHTU1plq6wLF1GslUAICm53kdkshgvZaUOvcBqpsNJrkrC7XsX9mAJqs= ceph-1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:25:00.630 INFO:teuthology.orchestra.run.smithi053.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:25:00.630 INFO:teuthology.orchestra.run.smithi053.stdout:Adding key to root@localhost authorized_keys... 2024-03-04T15:25:00.630 INFO:teuthology.orchestra.run.smithi053.stdout:Adding host smithi053... 2024-03-04T15:25:02.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:02 smithi053 ceph-mon[25738]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "addr": "172.21.15.53", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:02.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:02 smithi053 ceph-mon[25738]: Deploying cephadm binary to smithi053 2024-03-04T15:25:06.394 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Added host 'smithi053' with addr '172.21.15.53' 2024-03-04T15:25:06.394 INFO:teuthology.orchestra.run.smithi053.stdout:Deploying unmanaged mon service... 2024-03-04T15:25:07.104 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:06 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:25:07.104 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:06 smithi053 ceph-mon[25738]: Added host smithi053 2024-03-04T15:25:07.104 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:06 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:07.911 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-04T15:25:07.911 INFO:teuthology.orchestra.run.smithi053.stdout:Deploying unmanaged mgr service... 2024-03-04T15:25:08.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:08 smithi053 ceph-mon[25738]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:08.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:08 smithi053 ceph-mon[25738]: Saving service mon spec with placement count:5 2024-03-04T15:25:08.242 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:08 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:25:09.394 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-04T15:25:10.132 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:09 smithi053 ceph-mon[25738]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:10.133 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:09 smithi053 ceph-mon[25738]: Saving service mgr spec with placement count:2 2024-03-04T15:25:10.133 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:09 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:25:11.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:11 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1202680987' entity='client.admin' 2024-03-04T15:25:11.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:11 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:25:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:11 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:25:12.167 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling the dashboard module... 2024-03-04T15:25:12.592 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:12 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1743645011' entity='client.admin' 2024-03-04T15:25:13.858 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:13 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ignoring --setuser ceph since I am not root 2024-03-04T15:25:13.858 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:13 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ignoring --setgroup ceph since I am not root 2024-03-04T15:25:13.858 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:13 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/2480886707' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-04T15:25:13.858 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:13 smithi053 ceph-mon[25738]: from='mgr.14120 172.21.15.53:0/3756082016' entity='mgr.y' 2024-03-04T15:25:13.858 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:13 smithi053 ceph-mon[25738]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:14.135 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:14 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:14.096+0000 7f41038d5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:25:14.392 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:14 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:14.198+0000 7f41038d5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:25:14.874 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:14 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/2480886707' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-04T15:25:14.874 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:14 smithi053 ceph-mon[25738]: mgrmap e9: y(active, since 20s) 2024-03-04T15:25:15.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:14 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:14.871+0000 7f41038d5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:25:15.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:14 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:14.974+0000 7f41038d5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:25:15.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:15.117+0000 7f41038d5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:25:15.508 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:15.217+0000 7f41038d5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:25:15.509 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:15.302+0000 7f41038d5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:25:15.540 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:25:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-04T15:25:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:25:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-04T15:25:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:25:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:25:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for the mgr to restart... 2024-03-04T15:25:15.541 INFO:teuthology.orchestra.run.smithi053.stdout:Waiting for mgr epoch 9... 2024-03-04T15:25:15.782 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3208028763' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:25:15.783 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:15.578+0000 7f41038d5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:25:15.783 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:15.692+0000 7f41038d5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:25:15.783 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:15.780+0000 7f41038d5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:25:16.065 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:15.881+0000 7f41038d5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:25:16.065 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:16 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:16.062+0000 7f41038d5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:25:17.385 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:17 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:17.049+0000 7f41038d5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:25:17.386 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:17 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:17.183+0000 7f41038d5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:25:17.817 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:17 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:17.521+0000 7f41038d5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:25:17.817 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:17 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:17.614+0000 7f41038d5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:25:17.817 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:17 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:17.815+0000 7f41038d5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:25:18.135 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:18 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:18.004+0000 7f41038d5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:25:18.562 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:18 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:18.297+0000 7f41038d5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:25:18.888 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:18 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:18.559+0000 7f41038d5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:25:18.888 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:18 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:18.802+0000 7f41038d5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:25:19.885 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:19 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:19.627+0000 7f41038d5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:25:20.536 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:20.211+0000 7f41038d5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:25:20.537 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:20.313+0000 7f41038d5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:25:20.537 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:25:20.534+0000 7f41038d5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: Active manager daemon y restarted 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: Activating manager daemon y 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: osdmap e3: 0 total, 0 up, 0 in 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: mgrmap e10: y(active, starting, since 0.00333613s) 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: Manager daemon y is now available 2024-03-04T15:25:20.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:20.887 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: [04/Mar/2024:15:25:21] ENGINE Bus STARTING 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: [04/Mar/2024:15:25:21] ENGINE Serving on http://172.21.15.53:8765 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: [04/Mar/2024:15:25:21] ENGINE Serving on https://172.21.15.53:7150 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: [04/Mar/2024:15:25:21] ENGINE Bus STARTED 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: mgrmap e11: y(active, since 1.00588s) 2024-03-04T15:25:21.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:21 smithi053 ceph-mon[25738]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:25:22.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:25:22.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-04T15:25:22.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:25:22.607 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:25:22.607 INFO:teuthology.orchestra.run.smithi053.stdout:mgr epoch 9 is available 2024-03-04T15:25:22.608 INFO:teuthology.orchestra.run.smithi053.stdout:Generating a dashboard self-signed certificate... 2024-03-04T15:25:23.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:23.987 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-04T15:25:23.987 INFO:teuthology.orchestra.run.smithi053.stdout:Creating initial admin user... 2024-03-04T15:25:24.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:24 smithi053 ceph-mon[25738]: mgrmap e12: y(active, since 2s) 2024-03-04T15:25:24.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:24 smithi053 ceph-mon[25738]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:24.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:24 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:24.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:24 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:24.346 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:24 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:26.104 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:25 smithi053 ceph-mon[25738]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:26.105 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:26.107 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$yvPxvcDOScUMnwXwG.hfC.C77EP/Ez13pm1Gc6.Q7DLIN2k2WeCPy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709565924, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-04T15:25:26.107 INFO:teuthology.orchestra.run.smithi053.stdout:Fetching dashboard port number... 2024-03-04T15:25:27.051 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:26 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/726896120' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-04T15:25:27.313 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stdout 8443 2024-03-04T15:25:27.322 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:25:27.322 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout disabled 2024-03-04T15:25:27.329 INFO:teuthology.orchestra.run.smithi053.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:25:27.329 INFO:teuthology.orchestra.run.smithi053.stdout:systemctl: stdout inactive 2024-03-04T15:25:27.329 INFO:teuthology.orchestra.run.smithi053.stdout:firewalld.service is not enabled 2024-03-04T15:25:27.329 INFO:teuthology.orchestra.run.smithi053.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-04T15:25:27.330 INFO:teuthology.orchestra.run.smithi053.stdout:Ceph Dashboard is now available at: 2024-03-04T15:25:27.330 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:27.331 INFO:teuthology.orchestra.run.smithi053.stdout: URL: https://smithi053.front.sepia.ceph.com:8443/ 2024-03-04T15:25:27.331 INFO:teuthology.orchestra.run.smithi053.stdout: User: admin 2024-03-04T15:25:27.331 INFO:teuthology.orchestra.run.smithi053.stdout: Password: tghevhux01 2024-03-04T15:25:27.331 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:27.331 INFO:teuthology.orchestra.run.smithi053.stdout:Saving cluster configuration to /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config directory 2024-03-04T15:25:27.331 INFO:teuthology.orchestra.run.smithi053.stdout:Enabling autotune for osd_memory_target 2024-03-04T15:25:28.307 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:27 smithi053 ceph-mon[25738]: mgrmap e13: y(active, since 6s) 2024-03-04T15:25:29.899 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-04T15:25:29.899 INFO:teuthology.orchestra.run.smithi053.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout:Or, if you are only running a single cluster on this host: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: ceph telemetry on 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout:For more information see: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-04T15:25:29.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:29.901 INFO:teuthology.orchestra.run.smithi053.stdout:Bootstrap complete. 2024-03-04T15:25:29.947 INFO:tasks.cephadm:Fetching config... 2024-03-04T15:25:29.948 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:25:29.948 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-04T15:25:29.964 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-04T15:25:29.964 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:25:29.964 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-04T15:25:30.020 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-04T15:25:30.021 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:25:30.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-04T15:25:30.088 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-04T15:25:30.089 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:25:30.089 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-04T15:25:30.146 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-04T15:25:30.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6w2F8shCirH3X7cTPoWWHh0mC43mXuq+GFHjPhp1rrhOC92F6vnyCEb6dC3Gp4AthuARd3ftYIeMtKoIaHJVQyXxc5MMJIzoo6jHfgFKMc2zVJD1MbloKVUplJcMjzJn/0/MTG2AG9ynq4zx2LMEb2IQ4OHjHhBfG0Dv2/3voEhkqBB2B1B/jQm01BfOWYm/NpZ60mSIjmnfddKHeiqtCVVlXZGfOXoacm4hpXo2pQhb21b32te74SHCt7cB2IsQdJL6fAsAxGHAux/OwyDOe4nAIxik4GeilfO7xvwwosAa2YaRXNv5vp3jZjMMSCyEe5cHNk5pKyoIuMgqGYjrCHNRz+5Qcu5Q08eOVGVTKZ7L5MR3Rl9FvNe/VIB65oDHSf9B6RY/i8CL//hmj/rcp2UnGl1lyRLSj4YmPwSqYjS8AFq1g+YllaQf9jwU40gq92NioCWQEHTU1plq6wLF1GslUAICm53kdkshgvZaUOvcBqpsNJrkrC7XsX9mAJqs= ceph-1fe97ada-da3b-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:25:30.230 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6w2F8shCirH3X7cTPoWWHh0mC43mXuq+GFHjPhp1rrhOC92F6vnyCEb6dC3Gp4AthuARd3ftYIeMtKoIaHJVQyXxc5MMJIzoo6jHfgFKMc2zVJD1MbloKVUplJcMjzJn/0/MTG2AG9ynq4zx2LMEb2IQ4OHjHhBfG0Dv2/3voEhkqBB2B1B/jQm01BfOWYm/NpZ60mSIjmnfddKHeiqtCVVlXZGfOXoacm4hpXo2pQhb21b32te74SHCt7cB2IsQdJL6fAsAxGHAux/OwyDOe4nAIxik4GeilfO7xvwwosAa2YaRXNv5vp3jZjMMSCyEe5cHNk5pKyoIuMgqGYjrCHNRz+5Qcu5Q08eOVGVTKZ7L5MR3Rl9FvNe/VIB65oDHSf9B6RY/i8CL//hmj/rcp2UnGl1lyRLSj4YmPwSqYjS8AFq1g+YllaQf9jwU40gq92NioCWQEHTU1plq6wLF1GslUAICm53kdkshgvZaUOvcBqpsNJrkrC7XsX9mAJqs= ceph-1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:25:30.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6w2F8shCirH3X7cTPoWWHh0mC43mXuq+GFHjPhp1rrhOC92F6vnyCEb6dC3Gp4AthuARd3ftYIeMtKoIaHJVQyXxc5MMJIzoo6jHfgFKMc2zVJD1MbloKVUplJcMjzJn/0/MTG2AG9ynq4zx2LMEb2IQ4OHjHhBfG0Dv2/3voEhkqBB2B1B/jQm01BfOWYm/NpZ60mSIjmnfddKHeiqtCVVlXZGfOXoacm4hpXo2pQhb21b32te74SHCt7cB2IsQdJL6fAsAxGHAux/OwyDOe4nAIxik4GeilfO7xvwwosAa2YaRXNv5vp3jZjMMSCyEe5cHNk5pKyoIuMgqGYjrCHNRz+5Qcu5Q08eOVGVTKZ7L5MR3Rl9FvNe/VIB65oDHSf9B6RY/i8CL//hmj/rcp2UnGl1lyRLSj4YmPwSqYjS8AFq1g+YllaQf9jwU40gq92NioCWQEHTU1plq6wLF1GslUAICm53kdkshgvZaUOvcBqpsNJrkrC7XsX9mAJqs= ceph-1fe97ada-da3b-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:25:30.290 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6w2F8shCirH3X7cTPoWWHh0mC43mXuq+GFHjPhp1rrhOC92F6vnyCEb6dC3Gp4AthuARd3ftYIeMtKoIaHJVQyXxc5MMJIzoo6jHfgFKMc2zVJD1MbloKVUplJcMjzJn/0/MTG2AG9ynq4zx2LMEb2IQ4OHjHhBfG0Dv2/3voEhkqBB2B1B/jQm01BfOWYm/NpZ60mSIjmnfddKHeiqtCVVlXZGfOXoacm4hpXo2pQhb21b32te74SHCt7cB2IsQdJL6fAsAxGHAux/OwyDOe4nAIxik4GeilfO7xvwwosAa2YaRXNv5vp3jZjMMSCyEe5cHNk5pKyoIuMgqGYjrCHNRz+5Qcu5Q08eOVGVTKZ7L5MR3Rl9FvNe/VIB65oDHSf9B6RY/i8CL//hmj/rcp2UnGl1lyRLSj4YmPwSqYjS8AFq1g+YllaQf9jwU40gq92NioCWQEHTU1plq6wLF1GslUAICm53kdkshgvZaUOvcBqpsNJrkrC7XsX9mAJqs= ceph-1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:25:30.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6w2F8shCirH3X7cTPoWWHh0mC43mXuq+GFHjPhp1rrhOC92F6vnyCEb6dC3Gp4AthuARd3ftYIeMtKoIaHJVQyXxc5MMJIzoo6jHfgFKMc2zVJD1MbloKVUplJcMjzJn/0/MTG2AG9ynq4zx2LMEb2IQ4OHjHhBfG0Dv2/3voEhkqBB2B1B/jQm01BfOWYm/NpZ60mSIjmnfddKHeiqtCVVlXZGfOXoacm4hpXo2pQhb21b32te74SHCt7cB2IsQdJL6fAsAxGHAux/OwyDOe4nAIxik4GeilfO7xvwwosAa2YaRXNv5vp3jZjMMSCyEe5cHNk5pKyoIuMgqGYjrCHNRz+5Qcu5Q08eOVGVTKZ7L5MR3Rl9FvNe/VIB65oDHSf9B6RY/i8CL//hmj/rcp2UnGl1lyRLSj4YmPwSqYjS8AFq1g+YllaQf9jwU40gq92NioCWQEHTU1plq6wLF1GslUAICm53kdkshgvZaUOvcBqpsNJrkrC7XsX9mAJqs= ceph-1fe97ada-da3b-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:25:30.352 INFO:teuthology.orchestra.run.smithi160.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6w2F8shCirH3X7cTPoWWHh0mC43mXuq+GFHjPhp1rrhOC92F6vnyCEb6dC3Gp4AthuARd3ftYIeMtKoIaHJVQyXxc5MMJIzoo6jHfgFKMc2zVJD1MbloKVUplJcMjzJn/0/MTG2AG9ynq4zx2LMEb2IQ4OHjHhBfG0Dv2/3voEhkqBB2B1B/jQm01BfOWYm/NpZ60mSIjmnfddKHeiqtCVVlXZGfOXoacm4hpXo2pQhb21b32te74SHCt7cB2IsQdJL6fAsAxGHAux/OwyDOe4nAIxik4GeilfO7xvwwosAa2YaRXNv5vp3jZjMMSCyEe5cHNk5pKyoIuMgqGYjrCHNRz+5Qcu5Q08eOVGVTKZ7L5MR3Rl9FvNe/VIB65oDHSf9B6RY/i8CL//hmj/rcp2UnGl1lyRLSj4YmPwSqYjS8AFq1g+YllaQf9jwU40gq92NioCWQEHTU1plq6wLF1GslUAICm53kdkshgvZaUOvcBqpsNJrkrC7XsX9mAJqs= ceph-1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:25:30.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-04T15:25:30.612 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:25:30.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:30 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/929796976' entity='client.admin' 2024-03-04T15:25:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:33 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:33 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:33 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:25:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:33 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:33.323 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:33 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:33.324 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:33 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:33.679 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-04T15:25:33.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-04T15:25:33.923 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:25:34.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:34 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1648651520' entity='client.admin' 2024-03-04T15:25:34.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:34 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:34.279 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:34 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:36.450 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi064 2024-03-04T15:25:36.450 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:25:36.450 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/etc/ceph/ceph.conf 2024-03-04T15:25:36.469 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:25:36.469 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:25:36.526 INFO:tasks.cephadm:Adding host smithi064 to orchestrator... 2024-03-04T15:25:36.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch host add smithi064 2024-03-04T15:25:36.766 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:25:36.991 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:36.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:38.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:38 smithi053 ceph-mon[25738]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:39.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:39 smithi053 ceph-mon[25738]: Deploying cephadm binary to smithi064 2024-03-04T15:25:40.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:40 smithi053 ceph-mon[25738]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:42.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:42 smithi053 ceph-mon[25738]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:43.167 INFO:teuthology.orchestra.run.smithi053.stdout:Added host 'smithi064' with addr '172.21.15.64' 2024-03-04T15:25:43.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T15:25:43.904 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:25:44.338 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:44.338 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:44 smithi053 ceph-mon[25738]: Added host smithi064 2024-03-04T15:25:44.338 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:44.339 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:45.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:45 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:45.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:45 smithi053 ceph-mon[25738]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:45.771 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:45.771 INFO:teuthology.orchestra.run.smithi053.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}] 2024-03-04T15:25:46.345 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi160 2024-03-04T15:25:46.346 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:25:46.346 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.conf 2024-03-04T15:25:46.366 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:25:46.366 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:25:46.423 INFO:tasks.cephadm:Adding host smithi160 to orchestrator... 2024-03-04T15:25:46.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch host add smithi160 2024-03-04T15:25:46.664 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:25:47.046 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:46 smithi053 ceph-mon[25738]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:25:47.046 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:46 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:47.046 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:46 smithi053 ceph-mon[25738]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:48.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:48 smithi053 ceph-mon[25738]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:25:48.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:48 smithi053 ceph-mon[25738]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:50.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: Deploying cephadm binary to smithi160 2024-03-04T15:25:50.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:50.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:50.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:50.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:50.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:25:50.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:25:50.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:25:50.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:25:50.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:50 smithi053 ceph-mon[25738]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:25:51.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:51 smithi053 ceph-mon[25738]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:25:51.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:51 smithi053 ceph-mon[25738]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:52.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:52 smithi053 ceph-mon[25738]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:53.149 INFO:teuthology.orchestra.run.smithi053.stdout:Added host 'smithi160' with addr '172.21.15.160' 2024-03-04T15:25:53.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:25:53.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:53.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:53.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:53.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:53.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: Added host smithi160 2024-03-04T15:25:53.880 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:25:53.881 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:53 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:53.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T15:25:54.120 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:25:55.452 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:55.452 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:55 smithi053 ceph-mon[25738]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:55.870 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:25:55.871 INFO:teuthology.orchestra.run.smithi053.stdout:[{"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}, {"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.160", "hostname": "smithi160", "labels": [], "status": ""}] 2024-03-04T15:25:56.375 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-04T15:25:56.376 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-04T15:25:56.611 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:25:56.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:56 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:56.992 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:56 smithi053 ceph-mon[25738]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:25:56.993 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:56 smithi053 ceph-mon[25738]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-03-04T15:25:58.775 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:58 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/4002575345' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:25:59.063 INFO:tasks.cephadm:Adding mon.a on smithi053 2024-03-04T15:25:59.063 INFO:tasks.cephadm:Adding mon.b on smithi064 2024-03-04T15:25:59.063 INFO:tasks.cephadm:Adding mon.c on smithi160 2024-03-04T15:25:59.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c' 2024-03-04T15:25:59.296 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:25:59.342 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/4002575345' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:25:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:25:59 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:00.887 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:00 smithi053 ceph-mon[25738]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:26:00.887 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:00 smithi053 ceph-mon[25738]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:00.887 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:00 smithi053 ceph-mon[25738]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:00.887 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:00.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:00.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:01.173 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mon update... 2024-03-04T15:26:01.778 DEBUG:teuthology.orchestra.run.smithi064:mon.b> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.b.service 2024-03-04T15:26:01.780 DEBUG:teuthology.orchestra.run.smithi160:mon.c> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.c.service 2024-03-04T15:26:01.782 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:26:01.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:02.515 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:02.787 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:02 smithi053 ceph-mon[25738]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:02.787 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:02 smithi053 ceph-mon[25738]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c;count:3 2024-03-04T15:26:02.787 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:02 smithi053 ceph-mon[25738]: Deploying daemon mon.c on smithi160 2024-03-04T15:26:03.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:03 smithi053 ceph-mon[25738]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:04.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:04 smithi053 ceph-mon[25738]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:05.721 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 podman[30667]: 2024-03-04 15:26:05.6213765 +0000 UTC m=+0.019621964 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:26:05.721 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 podman[30667]: 2024-03-04 15:26:05.719402325 +0000 UTC m=+0.117647779 container create 19832f78866fd2d4f042dc4c27b9409452d56b1faaaad9dd321d403a07237f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-04T15:26:05.872 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:26:05.872 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":1,"fsid":"1fe97ada-da3b-11ee-95c6-87774f69a715","modified":"2024-03-04T15:24:22.928629Z","created":"2024-03-04T15:24:22.928629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:26:05.874 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 1 2024-03-04T15:26:06.157 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 podman[30667]: 2024-03-04 15:26:05.896549053 +0000 UTC m=+0.294794508 container init 19832f78866fd2d4f042dc4c27b9409452d56b1faaaad9dd321d403a07237f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:26:06.157 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 podman[30667]: 2024-03-04 15:26:05.902835369 +0000 UTC m=+0.301080831 container start 19832f78866fd2d4f042dc4c27b9409452d56b1faaaad9dd321d403a07237f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-04T15:26:06.157 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:26:06.157 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 7 2024-03-04T15:26:06.157 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pidfile_write: ignore empty --pid-file 2024-03-04T15:26:06.157 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: load: jerasure load: lrc 2024-03-04T15:26:06.157 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Git sha 0 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: DB SUMMARY 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: DB Session ID: YMSOYU6FX3GO64ITDN5A 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:26:06.158 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:26:06.159 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.env: 0x55774d156b20 2024-03-04T15:26:06.159 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:26:06.159 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.info_log: 0x55774f3938e0 2024-03-04T15:26:06.159 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:26:06.160 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.statistics: (nil) 2024-03-04T15:26:06.160 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.use_fsync: 0 2024-03-04T15:26:06.160 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:26:06.160 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:26:06.160 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:26:06.160 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.db_log_dir: 2024-03-04T15:26:06.161 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.wal_dir: 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.write_buffer_manager: 0x55774f367400 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:26:06.162 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.unordered_write: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.row_cache: None 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.wal_filter: None 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:26:06.163 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.wal_compression: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:26:06.164 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_open_files: -1 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:26:06.165 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Compression algorithms supported: 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kZSTD supported: 0 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:26:06.166 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.merge_operator: 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_filter: None 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55774f3939e0) 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout: index_type: 0 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout: data_block_index_type: 0 2024-03-04T15:26:06.167 INFO:journalctl@ceph.mon.c.smithi160.stdout: index_shortening: 1 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: checksum: 4 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: no_block_cache: 0 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_cache: 0x55774f399090 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_cache_options: 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: capacity : 536870912 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: num_shard_bits : 4 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: strict_capacity_limit : 0 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_cache_compressed: (nil) 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: persistent_cache: (nil) 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_size: 4096 2024-03-04T15:26:06.168 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_size_deviation: 10 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_restart_interval: 16 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: index_block_restart_interval: 1 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: metadata_block_size: 4096 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: partition_filters: 0 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: use_delta_encoding: 1 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: filter_policy: bloomfilter 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: whole_key_filtering: 1 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: verify_compression: 0 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: format_version: 5 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: enable_index_compression: 1 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: block_align: 0 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: max_auto_readahead_size: 262144 2024-03-04T15:26:06.169 INFO:journalctl@ceph.mon.c.smithi160.stdout: prepopulate_block_cache: 0 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression: NoCompression 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.num_levels: 7 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:26:06.170 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:26:06.171 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:26:06.172 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:26:06.173 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:26:06.174 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.ttl: 2592000 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:26:06.175 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 51ef7e46-bc56-4ef6-86e6-f6e6978e001c 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565965933497, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565965934089, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709565965, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "51ef7e46-bc56-4ef6-86e6-f6e6978e001c", "db_session_id": "YMSOYU6FX3GO64ITDN5A", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565965934195, "job": 1, "event": "recovery_finished"} 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55774f48c000 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: DB pointer 0x55774f476000 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:26:06.176 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: ** DB Stats ** 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: ** Compaction Stats [default] ** 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:06.177 INFO:journalctl@ceph.mon.c.smithi160.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: ** Compaction Stats [default] ** 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:26:06.178 INFO:journalctl@ceph.mon.c.smithi160.stdout: Cumulative compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: Interval compaction: 0.00 GB write, 0.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: Block cache BinnedLRUCache@0x55774f399090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: using public_addr v2:172.21.15.160:0/0 -> [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: starting mon.c rank -1 at public addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] at bind addrs [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(???) e0 preinit fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).mds e1 new map 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: e1 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:26:06.179 INFO:journalctl@ceph.mon.c.smithi160.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout: legacy client fscid: -1 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout: 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout: No filesystems configured 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:26:06.180 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:26:06.181 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.181 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.181 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:06.181 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.181 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Deploying cephadm binary to smithi160 2024-03-04T15:26:06.181 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.182 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Added host smithi160 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/4002575345' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/4002575345' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.183 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.184 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c;count:3 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: Deploying daemon mon.c on smithi160 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: from='client.? 172.21.15.160:0/1771698205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:05 smithi160 ceph-mon[30717]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:06 smithi160 bash[30667]: 19832f78866fd2d4f042dc4c27b9409452d56b1faaaad9dd321d403a07237f4f 2024-03-04T15:26:06.185 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:06 smithi160 systemd[1]: Started Ceph mon.c for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:26:06.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:05 smithi053 ceph-mon[25738]: from='client.? 172.21.15.160:0/1771698205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:07.593 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:26:07.593 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:26:07.837 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:26:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:26:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: mon.a calling monitor election 2024-03-04T15:26:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: mon.c calling monitor election 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: monmap e2: 2 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: fsmap 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: mgrmap e13: y(active, since 50s) 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: overall HEALTH_OK 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:11.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:11.388 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:11.388 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:11 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:26:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: mon.a calling monitor election 2024-03-04T15:26:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: mon.c calling monitor election 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: monmap e2: 2 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: fsmap 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: mgrmap e13: y(active, since 50s) 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: overall HEALTH_OK 2024-03-04T15:26:11.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:11.467 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:11.467 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:11.467 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:11.467 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:11 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:12.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:12 smithi053 ceph-mon[25738]: Deploying daemon mon.b on smithi064 2024-03-04T15:26:12.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:12 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:12.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:12 smithi160 ceph-mon[30717]: Deploying daemon mon.b on smithi064 2024-03-04T15:26:12.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:12 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:12.665 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:26:12.665 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":2,"fsid":"1fe97ada-da3b-11ee-95c6-87774f69a715","modified":"2024-03-04T15:26:05.960801Z","created":"2024-03-04T15:24:22.928629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:26:12.668 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 2 2024-03-04T15:26:13.244 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:13 smithi160 ceph-mon[30717]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:13.244 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:13 smithi160 ceph-mon[30717]: from='client.? 172.21.15.160:0/2141631695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:13.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:13 smithi053 ceph-mon[25738]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:13.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:13 smithi053 ceph-mon[25738]: from='client.? 172.21.15.160:0/2141631695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:14.245 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:26:14.245 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:26:14.491 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:26:14.724 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 podman[30258]: 2024-03-04 15:26:14.55180012 +0000 UTC m=+0.018647827 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:26:14.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:14 smithi053 ceph-mon[25738]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:14.922 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:14 smithi160 ceph-mon[30717]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 podman[30258]: 2024-03-04 15:26:14.810707073 +0000 UTC m=+0.277554773 container init 4526efd1013be8b832e1a305f2e6e4ba8ecb28e44145db8c8fb50dd9f2fc6783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 podman[30258]: 2024-03-04 15:26:14.815964307 +0000 UTC m=+0.282812005 container start 4526efd1013be8b832e1a305f2e6e4ba8ecb28e44145db8c8fb50dd9f2fc6783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222) 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 7 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pidfile_write: ignore empty --pid-file 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: load: jerasure load: lrc 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Git sha 0 2024-03-04T15:26:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: DB SUMMARY 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: DB Session ID: QJBMD5BKI7X15PBWONER 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:26:15.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.env: 0x55e15a8e9b20 2024-03-04T15:26:15.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:26:15.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.info_log: 0x55e15bf458e0 2024-03-04T15:26:15.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:26:15.275 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.statistics: (nil) 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.use_fsync: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:26:15.276 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.db_log_dir: 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.wal_dir: 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.write_buffer_manager: 0x55e15bf19400 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:26:15.277 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.unordered_write: 0 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.row_cache: None 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.wal_filter: None 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:26:15.278 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.wal_compression: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:26:15.279 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_open_files: -1 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:26:15.280 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Compression algorithms supported: 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kZSTD supported: 0 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-04T15:26:15.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.merge_operator: 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_filter: None 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e15bf459e0) 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:26:15.282 INFO:journalctl@ceph.mon.b.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: index_type: 0 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: data_block_index_type: 0 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: index_shortening: 1 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: checksum: 4 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: no_block_cache: 0 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache: 0x55e15bf4b090 2024-03-04T15:26:15.283 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache_options: 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: capacity : 536870912 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: num_shard_bits : 4 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: strict_capacity_limit : 0 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache_compressed: (nil) 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: persistent_cache: (nil) 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_size: 4096 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_size_deviation: 10 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_restart_interval: 16 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: index_block_restart_interval: 1 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: metadata_block_size: 4096 2024-03-04T15:26:15.284 INFO:journalctl@ceph.mon.b.smithi064.stdout: partition_filters: 0 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: use_delta_encoding: 1 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: filter_policy: bloomfilter 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: whole_key_filtering: 1 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: verify_compression: 0 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: format_version: 5 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: enable_index_compression: 1 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_align: 0 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: max_auto_readahead_size: 262144 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: prepopulate_block_cache: 0 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:26:15.285 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression: NoCompression 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.num_levels: 7 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:26:15.286 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:26:15.287 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:26:15.288 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:26:15.289 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:26:15.290 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.ttl: 2592000 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:26:15.291 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4ef21c73-5a2e-4aa2-9075-bc24209961d8 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565974845837, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565974846434, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709565974, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4ef21c73-5a2e-4aa2-9075-bc24209961d8", "db_session_id": "QJBMD5BKI7X15PBWONER", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709565974846542, "job": 1, "event": "recovery_finished"} 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e15c03e000 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: DB pointer 0x55e15c028000 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** DB Stats ** 2024-03-04T15:26:15.292 INFO:journalctl@ceph.mon.b.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** Compaction Stats [default] ** 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-03-04T15:26:15.293 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** Compaction Stats [default] ** 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:26:15.294 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: Block cache BinnedLRUCache@0x55e15bf4b090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: using public_addr v2:172.21.15.64:0/0 -> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: starting mon.b rank -1 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(???) e0 preinit fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:26:15.295 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).mds e1 new map 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout: e1 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout: legacy client fscid: -1 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout: No filesystems configured 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-04T15:26:15.296 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi160", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Deploying cephadm binary to smithi160 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.297 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Added host smithi160 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:15.298 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/4002575345' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/4002575345' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.299 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi160", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:15.300 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Saving service mon spec with placement smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c;count:3 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Deploying daemon mon.c on smithi160 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.? 172.21.15.160:0/1771698205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.a calling monitor election 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:15.301 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:15.302 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.c calling monitor election 2024-03-04T15:26:15.302 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: monmap e2: 2 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: fsmap 2024-03-04T15:26:15.303 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mgrmap e13: y(active, since 50s) 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: overall HEALTH_OK 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: Deploying daemon mon.b on smithi064 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: from='client.? 172.21.15.160:0/2141631695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 ceph-mon[30297]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-04T15:26:15.304 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 bash[30258]: 4526efd1013be8b832e1a305f2e6e4ba8ecb28e44145db8c8fb50dd9f2fc6783 2024-03-04T15:26:15.305 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:14 smithi064 systemd[1]: Started Ceph mon.b for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:26:20.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: mon.a calling monitor election 2024-03-04T15:26:20.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: mon.c calling monitor election 2024-03-04T15:26:20.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:26:20.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: fsmap 2024-03-04T15:26:20.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:20.217 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: mgrmap e13: y(active, since 59s) 2024-03-04T15:26:20.217 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:26:20.217 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:26:20.217 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:26:20.217 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: mon.b (rank 2) addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] is down (out of quorum) 2024-03-04T15:26:20.217 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:20.217 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:20.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: mon.a calling monitor election 2024-03-04T15:26:20.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: mon.c calling monitor election 2024-03-04T15:26:20.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:26:20.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:20.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: fsmap 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: mgrmap e13: y(active, since 59s) 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: mon.b (rank 2) addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] is down (out of quorum) 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:20.387 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:20.622 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-03-04T15:26:20.622 INFO:teuthology.orchestra.run.smithi160.stdout:{"epoch":3,"fsid":"1fe97ada-da3b-11ee-95c6-87774f69a715","modified":"2024-03-04T15:26:14.871752Z","created":"2024-03-04T15:24:22.928629Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:3300","nonce":0},{"type":"v1","addr":"172.21.15.160:6789","nonce":0}]},"addr":"172.21.15.160:6789/0","public_addr":"172.21.15.160:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:26:20.624 INFO:teuthology.orchestra.run.smithi160.stderr:dumped monmap epoch 3 2024-03-04T15:26:21.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:21.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:21.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:21.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:20 smithi160 ceph-mon[30717]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:21.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:21.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:20 smithi160 ceph-mon[30717]: from='client.? 172.21.15.160:0/3369811841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:21.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:21.255 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-04T15:26:21.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-04T15:26:21.285 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:21.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:21.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:21.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:20 smithi053 ceph-mon[25738]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:21.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:21.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:20 smithi053 ceph-mon[25738]: from='client.? 172.21.15.160:0/3369811841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:21.286 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:21.510 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:26:22.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: mon.a calling monitor election 2024-03-04T15:26:22.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: mon.c calling monitor election 2024-03-04T15:26:22.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:26:22.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:22.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:26:22.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: fsmap 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: mgrmap e13: y(active, since 59s) 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-04T15:26:22.273 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: mon.b (rank 2) addr [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] is down (out of quorum) 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='client.? 172.21.15.160:0/3369811841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:26:22.274 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:22.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: mon.b calling monitor election 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: mon.b calling monitor election 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: mon.a calling monitor election 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: mon.c calling monitor election 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: fsmap 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: mgrmap e13: y(active, since 61s) 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: Cluster is now healthy 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: overall HEALTH_OK 2024-03-04T15:26:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:22.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:22 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: mon.b calling monitor election 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: mon.b calling monitor election 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: mon.a calling monitor election 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: mon.c calling monitor election 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: fsmap 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: mgrmap e13: y(active, since 61s) 2024-03-04T15:26:22.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:26:22.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: Cluster is now healthy 2024-03-04T15:26:22.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: overall HEALTH_OK 2024-03-04T15:26:22.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:22.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:22 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:22.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: mon.b calling monitor election 2024-03-04T15:26:22.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: mon.b calling monitor election 2024-03-04T15:26:22.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: mon.a calling monitor election 2024-03-04T15:26:22.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: mon.c calling monitor election 2024-03-04T15:26:22.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-04T15:26:22.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: monmap e3: 3 mons at {a=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: fsmap 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: mgrmap e13: y(active, since 61s) 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: Cluster is now healthy 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: overall HEALTH_OK 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:22.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:22 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:23.482 INFO:teuthology.orchestra.run.smithi053.stdout:# minimal ceph.conf for 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:26:23.482 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-03-04T15:26:23.482 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:26:23.482 INFO:teuthology.orchestra.run.smithi053.stdout: mon_host = [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.160:3300/0,v1:172.21.15.160:6789/0] 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:23.875 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:23.876 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/490938328' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:23.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-04T15:26:23.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:26:23.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:26:23.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:23.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:23.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:23.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:23.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:23.967 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:23 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/490938328' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:26:24.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:23 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/490938328' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:24.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:26:23 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:26:23.871+0000 7f40b9ac8700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-04T15:26:24.278 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-04T15:26:24.279 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:26:24.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:26:24.350 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:26:24.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:24.419 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:26:24.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:26:24.454 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:26:24.455 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:24.525 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:26:24.526 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:26:24.561 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:26:24.561 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:26:24.632 INFO:tasks.cephadm:Adding mgr.y on smithi053 2024-03-04T15:26:24.632 INFO:tasks.cephadm:Adding mgr.x on smithi064 2024-03-04T15:26:24.632 INFO:tasks.cephadm:Adding mgr.z on smithi160 2024-03-04T15:26:24.632 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch apply mgr '3;smithi053=y;smithi064=x;smithi160=z' 2024-03-04T15:26:24.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:24 smithi064 ceph-mon[30297]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T15:26:24.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:24 smithi064 ceph-mon[30297]: Reconfiguring daemon mon.a on smithi053 2024-03-04T15:26:24.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:24 smithi053 ceph-mon[25738]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T15:26:24.836 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:24 smithi053 ceph-mon[25738]: Reconfiguring daemon mon.a on smithi053 2024-03-04T15:26:24.903 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:26:24.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:24 smithi160 ceph-mon[30717]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T15:26:24.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:24 smithi160 ceph-mon[30717]: Reconfiguring daemon mon.a on smithi053 2024-03-04T15:26:25.691 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:25 smithi160 ceph-mon[30717]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:25.691 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:25.691 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:25.691 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:25.691 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:25.691 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:25.799 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:25 smithi064 ceph-mon[30297]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:25.800 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:25.800 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:25.800 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:25.800 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:25.800 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:25.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:25 smithi053 ceph-mon[25738]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:25.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:25.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:25.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:25.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:25.888 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:26.667 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled mgr update... 2024-03-04T15:26:26.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:26 smithi064 ceph-mon[30297]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:26:26.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:26 smithi064 ceph-mon[30297]: Reconfiguring daemon mon.b on smithi064 2024-03-04T15:26:26.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:26 smithi053 ceph-mon[25738]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:26:26.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:26 smithi053 ceph-mon[25738]: Reconfiguring daemon mon.b on smithi064 2024-03-04T15:26:26.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:26 smithi160 ceph-mon[30717]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:26:26.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:26 smithi160 ceph-mon[30717]: Reconfiguring daemon mon.b on smithi064 2024-03-04T15:26:27.342 DEBUG:teuthology.orchestra.run.smithi064:mgr.x> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.x.service 2024-03-04T15:26:27.344 DEBUG:teuthology.orchestra.run.smithi160:mgr.z> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.z.service 2024-03-04T15:26:27.347 INFO:tasks.cephadm:Deploying OSDs... 2024-03-04T15:26:27.347 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:26:27.347 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:27.366 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:26:27.366 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/nvme1n1 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/nvme1n1 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:25:30.820392882 +0000 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:23:12.635100719 +0000 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:23:12.635100719 +0000 2024-03-04T15:26:27.425 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:23:12.624100927 +0000 2024-03-04T15:26:27.426 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-04T15:26:27.495 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:26:27.495 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:26:27.495 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000168758 s, 3.0 MB/s 2024-03-04T15:26:27.496 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-04T15:26:27.555 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/nvme2n1 2024-03-04T15:26:27.606 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:27 smithi160 ceph-mon[30717]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:27 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:27 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:27 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:27 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:27 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:27.607 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:27 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/nvme2n1 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:25:30.829392713 +0000 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:23:12.785097894 +0000 2024-03-04T15:26:27.612 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:23:12.785097894 +0000 2024-03-04T15:26:27.613 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:23:12.776098064 +0000 2024-03-04T15:26:27.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-04T15:26:27.680 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:26:27.680 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:26:27.680 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000128088 s, 4.0 MB/s 2024-03-04T15:26:27.681 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-04T15:26:27.739 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/nvme3n1 2024-03-04T15:26:27.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:27 smithi064 ceph-mon[30297]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:27 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:27 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:27 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:27 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:27 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:27 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:27.794 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/nvme3n1 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:25:30.837392563 +0000 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:23:12.929095182 +0000 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:23:12.929095182 +0000 2024-03-04T15:26:27.795 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:23:12.920095351 +0000 2024-03-04T15:26:27.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-04T15:26:27.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:27 smithi053 ceph-mon[25738]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:27.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:27 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:27 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:27 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:27.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:27 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:26:27.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:27 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:26:27.860 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:27 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:27.862 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:26:27.862 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:26:27.862 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000136876 s, 3.7 MB/s 2024-03-04T15:26:27.863 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-04T15:26:27.920 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/nvme4n1 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/nvme4n1 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-04 15:25:30.847392375 +0000 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-04 15:23:13.064092639 +0000 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-04 15:23:13.064092639 +0000 2024-03-04T15:26:27.977 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-04 15:23:13.063092658 +0000 2024-03-04T15:26:27.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-04T15:26:28.044 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-04T15:26:28.044 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-04T15:26:28.044 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.00013352 s, 3.8 MB/s 2024-03-04T15:26:28.045 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-04T15:26:28.102 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:26:28.102 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:28.121 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:26:28.122 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme1n1 2024-03-04T15:26:28.179 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme1n1 2024-03-04T15:26:28.179 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:28.179 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-04T15:26:28.179 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:28.179 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:28.180 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:25:48.190266246 +0000 2024-03-04T15:26:28.180 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:23:14.282230412 +0000 2024-03-04T15:26:28.180 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:23:14.282230412 +0000 2024-03-04T15:26:28.180 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:23:14.271236537 +0000 2024-03-04T15:26:28.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-04T15:26:28.249 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:26:28.250 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:26:28.250 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000245914 s, 2.1 MB/s 2024-03-04T15:26:28.251 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-04T15:26:28.309 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme2n1 2024-03-04T15:26:28.366 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme2n1 2024-03-04T15:26:28.366 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:28.366 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-04T15:26:28.366 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:28.366 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:28.367 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:25:48.201266043 +0000 2024-03-04T15:26:28.367 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:23:14.437144111 +0000 2024-03-04T15:26:28.367 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:23:14.437144111 +0000 2024-03-04T15:26:28.367 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:23:14.429148565 +0000 2024-03-04T15:26:28.367 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-04T15:26:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:26:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:26:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000241918 s, 2.1 MB/s 2024-03-04T15:26:28.436 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-04T15:26:28.494 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme3n1 2024-03-04T15:26:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme3n1 2024-03-04T15:26:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:28.550 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-04T15:26:28.550 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:28.550 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:28.550 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:25:48.212265840 +0000 2024-03-04T15:26:28.551 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:23:14.584136664 +0000 2024-03-04T15:26:28.551 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:23:14.584136664 +0000 2024-03-04T15:26:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:23:14.573136864 +0000 2024-03-04T15:26:28.551 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-04T15:26:28.617 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:26:28.618 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:26:28.618 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000316642 s, 1.6 MB/s 2024-03-04T15:26:28.619 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-04T15:26:28.675 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme4n1 2024-03-04T15:26:28.730 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme4n1 2024-03-04T15:26:28.730 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:28.730 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-04T15:26:28.730 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:28.730 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:28.730 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-03-04 15:25:48.223265637 +0000 2024-03-04T15:26:28.731 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-03-04 15:23:14.728134055 +0000 2024-03-04T15:26:28.731 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-03-04 15:23:14.728134055 +0000 2024-03-04T15:26:28.731 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-03-04 15:23:14.717134255 +0000 2024-03-04T15:26:28.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-04T15:26:28.796 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-03-04T15:26:28.796 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-03-04T15:26:28.797 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000220556 s, 2.3 MB/s 2024-03-04T15:26:28.797 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:28 smithi064 ceph-mon[30297]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=y;smithi064=x;smithi160=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:28.797 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:28 smithi064 ceph-mon[30297]: Saving service mgr spec with placement smithi053=y;smithi064=x;smithi160=z;count:3 2024-03-04T15:26:28.797 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:28 smithi064 ceph-mon[30297]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:26:28.797 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:28 smithi064 ceph-mon[30297]: Reconfiguring daemon mon.c on smithi160 2024-03-04T15:26:28.797 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:28 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:28.797 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:28 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:28.798 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:28 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:28.798 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-04T15:26:28.855 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:26:28.855 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:26:28.871 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-04T15:26:28.871 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/nvme1n1 2024-03-04T15:26:28.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:28 smithi053 ceph-mon[25738]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=y;smithi064=x;smithi160=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:28.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:28 smithi053 ceph-mon[25738]: Saving service mgr spec with placement smithi053=y;smithi064=x;smithi160=z;count:3 2024-03-04T15:26:28.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:28 smithi053 ceph-mon[25738]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:26:28.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:28 smithi053 ceph-mon[25738]: Reconfiguring daemon mon.c on smithi160 2024-03-04T15:26:28.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:28 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:28.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:28 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:28.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:28 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:28.926 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/nvme1n1 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 763 Links: 1 Device type: 103,2 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:25:57.970042797 +0000 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:23:15.948154378 +0000 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:23:15.948154378 +0000 2024-03-04T15:26:28.927 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:23:15.940154482 +0000 2024-03-04T15:26:28.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-04T15:26:28.968 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:28 smithi160 ceph-mon[30717]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi053=y;smithi064=x;smithi160=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:28.968 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:28 smithi160 ceph-mon[30717]: Saving service mgr spec with placement smithi053=y;smithi064=x;smithi160=z;count:3 2024-03-04T15:26:28.968 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:28 smithi160 ceph-mon[30717]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:26:28.969 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:28 smithi160 ceph-mon[30717]: Reconfiguring daemon mon.c on smithi160 2024-03-04T15:26:28.969 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:28 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:28.969 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:28 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:28.969 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:28 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:28.995 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:26:28.996 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:26:28.996 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000229242 s, 2.2 MB/s 2024-03-04T15:26:28.997 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-04T15:26:29.053 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/nvme2n1 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/nvme2n1 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 776 Links: 1 Device type: 103,4 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:25:57.982042641 +0000 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:23:16.100152401 +0000 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:23:16.100152401 +0000 2024-03-04T15:26:29.109 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:23:16.088152557 +0000 2024-03-04T15:26:29.110 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-04T15:26:29.176 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:26:29.176 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:26:29.176 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000251934 s, 2.0 MB/s 2024-03-04T15:26:29.177 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-04T15:26:29.234 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/nvme3n1 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/nvme3n1 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:25:57.993042498 +0000 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:23:16.244150529 +0000 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:23:16.244150529 +0000 2024-03-04T15:26:29.289 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:23:16.233150672 +0000 2024-03-04T15:26:29.290 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-04T15:26:29.355 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:26:29.355 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:26:29.355 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000212879 s, 2.4 MB/s 2024-03-04T15:26:29.356 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-04T15:26:29.413 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/nvme4n1 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/nvme4n1 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-03-04 15:25:58.004042356 +0000 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-03-04 15:23:16.395148565 +0000 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-03-04 15:23:16.395148565 +0000 2024-03-04T15:26:29.468 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: 2024-03-04 15:23:16.388148656 +0000 2024-03-04T15:26:29.469 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-04T15:26:29.533 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-03-04T15:26:29.533 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-03-04T15:26:29.534 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000208742 s, 2.5 MB/s 2024-03-04T15:26:29.534 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-04T15:26:29.590 INFO:tasks.cephadm:Deploying osd.0 on smithi053 with /dev/nvme4n1... 2024-03-04T15:26:29.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-04T15:26:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:26:29.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:29.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:29 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:29.825 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:29.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:29 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:30.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:29 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:30.668 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:30 smithi160 ceph-mon[30717]: Deploying daemon mgr.z on smithi160 2024-03-04T15:26:30.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:30 smithi053 ceph-mon[25738]: Deploying daemon mgr.z on smithi160 2024-03-04T15:26:31.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:30 smithi064 ceph-mon[30297]: Deploying daemon mgr.z on smithi160 2024-03-04T15:26:31.347 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:31 smithi160 podman[32785]: 2024-03-04 15:26:31.236165113 +0000 UTC m=+0.124330975 container create 89ece9067207bb23fe44014185f691d23794058a86d6397f201db283d9f767c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-04T15:26:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:31 smithi160 ceph-mon[30717]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:31.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:31 smithi160 podman[32785]: 2024-03-04 15:26:31.455115036 +0000 UTC m=+0.343280869 container init 89ece9067207bb23fe44014185f691d23794058a86d6397f201db283d9f767c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:26:31.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:31 smithi160 podman[32785]: 2024-03-04 15:26:31.460162334 +0000 UTC m=+0.348328155 container start 89ece9067207bb23fe44014185f691d23794058a86d6397f201db283d9f767c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-03-04T15:26:31.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:31 smithi160 ceph-mgr[32823]: -- 172.21.15.160:0/3811215261 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e42452e5a0 con 0x55e424503000 2024-03-04T15:26:31.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:31 smithi160 bash[32785]: 89ece9067207bb23fe44014185f691d23794058a86d6397f201db283d9f767c9 2024-03-04T15:26:31.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:31 smithi160 systemd[1]: Started Ceph mgr.z for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:26:31.729 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping: /dev/nvme4n1 2024-03-04T15:26:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-04T15:26:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-04T15:26:31.730 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10+0 records in 2024-03-04T15:26:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:10+0 records out 2024-03-04T15:26:31.730 INFO:teuthology.orchestra.run.smithi053.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0103201 s, 1.0 GB/s 2024-03-04T15:26:31.730 INFO:teuthology.orchestra.run.smithi053.stderr:--> Zapping successful for: 2024-03-04T15:26:31.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:31 smithi053 ceph-mon[25738]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:32.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:31 smithi064 ceph-mon[30297]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:32.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi053:/dev/nvme4n1 2024-03-04T15:26:32.465 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:31.999+0000 7fdca624b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:26:32.465 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:32.089+0000 7fdca624b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:26:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:26:32.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:32.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:26:32.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:32.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:32.831 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: Deploying daemon mgr.x on smithi064 2024-03-04T15:26:32.831 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:32 smithi053 ceph-mon[25738]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:32.876 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.876 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.876 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.877 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:32.877 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:26:32.877 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:32.877 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:32.877 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: Deploying daemon mgr.x on smithi064 2024-03-04T15:26:32.877 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:32 smithi064 ceph-mon[30297]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:32.965 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:32.727+0000 7fdca624b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:26:32.965 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:32.823+0000 7fdca624b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:26:32.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:32.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:32.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:26:32.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:32.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:32.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: Deploying daemon mgr.x on smithi064 2024-03-04T15:26:32.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-mon[30717]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:33.431 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:32 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:32.972+0000 7fdca624b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:26:33.432 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:33 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:33.072+0000 7fdca624b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:26:33.432 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:33 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:33.158+0000 7fdca624b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:26:33.432 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:33 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:33.429+0000 7fdca624b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:26:33.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:33 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:33.541+0000 7fdca624b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:26:33.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:33 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:33.629+0000 7fdca624b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:26:33.935 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:33 smithi064 systemd[1]: Starting Ceph mgr.x for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:26:34.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:33 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:33.725+0000 7fdca624b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:26:34.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:33 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:33.905+0000 7fdca624b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:26:34.271 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:33 smithi064 podman[31745]: 2024-03-04 15:26:33.836699993 +0000 UTC m=+0.016785574 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:26:34.272 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:33 smithi064 podman[31745]: 2024-03-04 15:26:33.944209457 +0000 UTC m=+0.124295065 container create 4fbf03a2016d5f2dde4dc8105c69ba1f94f0c678451ca75c8ca1945a15fb43f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:26:34.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:34 smithi064 podman[31745]: 2024-03-04 15:26:34.321660963 +0000 UTC m=+0.501746537 container init 4fbf03a2016d5f2dde4dc8105c69ba1f94f0c678451ca75c8ca1945a15fb43f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:26:34.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:34 smithi064 podman[31745]: 2024-03-04 15:26:34.3270222 +0000 UTC m=+0.507107776 container start 4fbf03a2016d5f2dde4dc8105c69ba1f94f0c678451ca75c8ca1945a15fb43f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-04T15:26:34.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:34 smithi064 ceph-mgr[31785]: -- 172.21.15.64:0/1433067737 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b7b79cc5a0 con 0x55b7b79a0800 2024-03-04T15:26:34.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:34 smithi064 bash[31745]: 4fbf03a2016d5f2dde4dc8105c69ba1f94f0c678451ca75c8ca1945a15fb43f8 2024-03-04T15:26:34.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:34 smithi064 systemd[1]: Started Ceph mgr.x for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:26:35.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:34 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:34.895+0000 7fdca624b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:26:35.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:35.031+0000 7fdca624b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:26:35.271 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:34 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:34.906+0000 7f09cba93200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:26:35.272 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:35.005+0000 7f09cba93200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:26:35.609 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.609 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.609 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.609 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.609 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:35.610 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:35.610 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:26:35.610 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:26:35.610 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:35 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:35.662 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.662 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.662 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.662 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.662 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:35.662 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:35.663 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:26:35.663 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:26:35.663 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:35.395+0000 7fdca624b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:35.483+0000 7fdca624b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:35.669 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:35.670 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:26:35.670 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:26:35.670 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:35.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:35.712+0000 7f09cba93200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:26:35.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:35.810+0000 7f09cba93200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:26:35.965 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:35.694+0000 7fdca624b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:26:35.965 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:35 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:35.878+0000 7fdca624b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:26:36.271 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:35 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:35.952+0000 7f09cba93200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:26:36.272 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:36.051+0000 7f09cba93200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:26:36.272 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:36.135+0000 7f09cba93200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:26:36.436 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:36.169+0000 7fdca624b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:26:36.436 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:36.433+0000 7fdca624b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:26:36.712 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:36.411+0000 7f09cba93200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:26:36.712 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:36.523+0000 7f09cba93200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:26:36.712 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:36.612+0000 7f09cba93200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:26:36.712 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:36.709+0000 7f09cba93200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:36.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:36.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:36.714 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:36.667+0000 7fdca624b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:36.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:36.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:36.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:36 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:36.728 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:26:36.728 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.728 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.728 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.728 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.728 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:36.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:36.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:36.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:26:36.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:26:36.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:36 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:37.021 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:36 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:36.898+0000 7f09cba93200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:37.452+0000 7fdca624b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-mon[30717]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-mon[30717]: Reconfiguring daemon mgr.y on smithi053 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-mon[30717]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1257792859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c0477ff-d13b-4394-902f-7800c9374c58"}]: dispatch 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1257792859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c0477ff-d13b-4394-902f-7800c9374c58"}]': finished 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-mon[30717]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:26:37.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:37 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:37.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:37 smithi053 ceph-mon[25738]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:26:37.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:37 smithi053 ceph-mon[25738]: Reconfiguring daemon mgr.y on smithi053 2024-03-04T15:26:37.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:37 smithi053 ceph-mon[25738]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:37.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:37 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1257792859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c0477ff-d13b-4394-902f-7800c9374c58"}]: dispatch 2024-03-04T15:26:37.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:37 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1257792859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c0477ff-d13b-4394-902f-7800c9374c58"}]': finished 2024-03-04T15:26:37.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:37 smithi053 ceph-mon[25738]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:26:37.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:37 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:37.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-mon[30297]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:26:37.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-mon[30297]: Reconfiguring daemon mgr.y on smithi053 2024-03-04T15:26:37.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-mon[30297]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:37.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1257792859' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c0477ff-d13b-4394-902f-7800c9374c58"}]: dispatch 2024-03-04T15:26:37.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1257792859' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c0477ff-d13b-4394-902f-7800c9374c58"}]': finished 2024-03-04T15:26:37.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-mon[30297]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:26:37.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:38.271 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:37 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:37.893+0000 7f09cba93200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:26:38.272 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:38.027+0000 7f09cba93200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:26:38.337 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:38.023+0000 7fdca624b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:26:38.337 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:38.120+0000 7fdca624b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:26:38.337 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:26:38.334+0000 7fdca624b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:26:38.672 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-mon[30297]: Standby manager daemon z started 2024-03-04T15:26:38.673 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:26:38.673 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:26:38.673 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:26:38.673 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:26:38.673 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:38.382+0000 7f09cba93200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:26:38.673 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:38.473+0000 7f09cba93200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:26:38.673 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:38.670+0000 7f09cba93200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:26:38.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-mon[30717]: Standby manager daemon z started 2024-03-04T15:26:38.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:26:38.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:26:38.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:26:38.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:38 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:26:38.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:38 smithi053 ceph-mon[25738]: Standby manager daemon z started 2024-03-04T15:26:38.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:38 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:26:38.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:38 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:26:38.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:38 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:26:38.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:38 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/2572019856' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:26:39.021 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:38 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:38.858+0000 7f09cba93200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:26:39.408 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:39 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:39.146+0000 7f09cba93200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:26:39.408 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:39 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:39.405+0000 7f09cba93200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:26:39.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:39 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:39.645+0000 7f09cba93200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:26:39.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:39 smithi064 ceph-mon[30297]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-04T15:26:39.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:39 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:26:39.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:39 smithi064 ceph-mon[30297]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:39.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:39 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3544152269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:26:39.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:39 smithi053 ceph-mon[25738]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-04T15:26:39.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:39 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:26:39.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:39 smithi053 ceph-mon[25738]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:39.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:39 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3544152269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:26:39.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:39 smithi160 ceph-mon[30717]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-04T15:26:39.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:39 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:26:39.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:39 smithi160 ceph-mon[30717]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:39.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:39 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3544152269' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:26:40.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:40 smithi064 ceph-mon[30297]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:40.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:40 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:40.465+0000 7f09cba93200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:26:40.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:40 smithi053 ceph-mon[25738]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:40.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:40 smithi160 ceph-mon[30717]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:41.362 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:41.043+0000 7f09cba93200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:26:41.362 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:41.142+0000 7f09cba93200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:26:41.362 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:26:41.360+0000 7f09cba93200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:26:41.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:41 smithi053 ceph-mon[25738]: Standby manager daemon x started 2024-03-04T15:26:41.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:41 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:26:41.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:41 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:26:41.841 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:41 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:26:41.842 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:41 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:26:41.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:41 smithi160 ceph-mon[30717]: Standby manager daemon x started 2024-03-04T15:26:41.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:41 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:26:41.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:41 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:26:41.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:41 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:26:41.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:41 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:26:42.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-mon[30297]: Standby manager daemon x started 2024-03-04T15:26:42.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:26:42.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:26:42.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:26:42.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:41 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/1737555462' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:26:42.774 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:42 smithi053 ceph-mon[25738]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-04T15:26:42.774 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:42 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:26:42.775 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:42 smithi053 ceph-mon[25738]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:42.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:42 smithi160 ceph-mon[30717]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-04T15:26:42.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:42 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:26:42.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:42 smithi160 ceph-mon[30717]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:43.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:42 smithi064 ceph-mon[30297]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-04T15:26:43.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:42 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:26:43.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:42 smithi064 ceph-mon[30297]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:45.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:45.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:45.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:45.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:44 smithi053 ceph-mon[25738]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:45.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:45.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:45.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:45.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:44 smithi160 ceph-mon[30717]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:45.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:45.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:45.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:45.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:45.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:44 smithi064 ceph-mon[30297]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:46.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:45 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:26:46.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:45 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:46.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:45 smithi053 ceph-mon[25738]: Deploying daemon osd.0 on smithi053 2024-03-04T15:26:46.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:45 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:26:46.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:45 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:46.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:45 smithi160 ceph-mon[30717]: Deploying daemon osd.0 on smithi053 2024-03-04T15:26:46.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:45 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:26:46.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:45 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:46.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:45 smithi064 ceph-mon[30297]: Deploying daemon osd.0 on smithi053 2024-03-04T15:26:46.936 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:46 smithi053 ceph-mon[25738]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:47.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:46 smithi160 ceph-mon[30717]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:47.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:46 smithi064 ceph-mon[30297]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:48.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:48 smithi053 ceph-mon[25738]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:48.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:48 smithi160 ceph-mon[30717]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:49.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:48 smithi064 ceph-mon[30297]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:50.756 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:50 smithi053 ceph-mon[25738]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:50 smithi160 ceph-mon[30717]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:51.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:50 smithi064 ceph-mon[30297]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:52.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:52 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:52.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:52 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:52.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:52 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:52.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:52 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:52.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:52 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:52.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:52 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:52.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:52 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:52.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:52 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:52.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:52 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:53.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:53 smithi053 ceph-mon[25738]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:53.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:53 smithi064 ceph-mon[30297]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:53.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:53 smithi160 ceph-mon[30717]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:54.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:54 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:54.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:54 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:54.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:54 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:54.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:54 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:54.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:54 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:54.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:54 smithi053 ceph-mon[25738]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:26:54.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:54 smithi053 ceph-mon[25738]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:54.952 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 0 on host 'smithi053' 2024-03-04T15:26:54.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:54 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:54.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:54 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:54.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:54 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:54.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:54 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:54.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:54 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:54.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:54 smithi160 ceph-mon[30717]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:26:54.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:54 smithi160 ceph-mon[30717]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:55.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:54 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:55.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:54 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:55.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:54 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:26:55.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:54 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:26:55.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:54 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:55.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:54 smithi064 ceph-mon[30297]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:26:55.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:54 smithi064 ceph-mon[30297]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:55.658 DEBUG:teuthology.orchestra.run.smithi053:osd.0> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.0.service 2024-03-04T15:26:55.660 INFO:tasks.cephadm:Deploying osd.1 on smithi064 with /dev/nvme4n1... 2024-03-04T15:26:55.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-04T15:26:55.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:55 smithi053 ceph-mon[25738]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:26:55.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:55 smithi053 ceph-mon[25738]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:26:55.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:55 smithi053 ceph-mon[25738]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-04T15:26:55.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:55.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:55.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:55.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:55.911 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:26:56.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:55 smithi064 ceph-mon[30297]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:26:56.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:55 smithi064 ceph-mon[30297]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:26:56.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:55 smithi064 ceph-mon[30297]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-04T15:26:56.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:56.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:56.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:56.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:56.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:55 smithi160 ceph-mon[30717]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:26:56.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:55 smithi160 ceph-mon[30717]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:26:56.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:55 smithi160 ceph-mon[30717]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-03-04T15:26:56.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:56.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:56.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:26:56.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:26:56.218 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0[40429]: 2024-03-04T15:26:56.007+0000 7f3438810700 -1 osd.0 0 waiting for initial osdmap 2024-03-04T15:26:56.218 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0[40429]: 2024-03-04T15:26:56.009+0000 7f34305fb700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:26:56.975 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:56 smithi064 ceph-mon[30297]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-04T15:26:56.975 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:56 smithi064 ceph-mon[30297]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:26:56.975 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:56 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:56.975 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:56 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:56.975 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:56 smithi064 ceph-mon[30297]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' 2024-03-04T15:26:56.975 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:56 smithi064 ceph-mon[30297]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:57.012 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-mon[25738]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-04T15:26:57.013 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-mon[25738]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:26:57.013 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:57.013 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:57.013 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-mon[25738]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' 2024-03-04T15:26:57.013 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:56 smithi053 ceph-mon[25738]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:57.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:56 smithi160 ceph-mon[30717]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-03-04T15:26:57.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:56 smithi160 ceph-mon[30717]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:26:57.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:56 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:57.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:56 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:57.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:56 smithi160 ceph-mon[30717]: from='osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209]' entity='osd.0' 2024-03-04T15:26:57.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:56 smithi160 ceph-mon[30717]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:26:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/nvme4n1 2024-03-04T15:26:57.804 INFO:teuthology.orchestra.run.smithi064.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-04T15:26:57.805 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-04T15:26:57.805 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-03-04T15:26:57.805 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-03-04T15:26:57.805 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.017122 s, 612 MB/s 2024-03-04T15:26:57.805 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-03-04T15:26:58.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:57 smithi064 ceph-mon[30297]: purged_snaps scrub starts 2024-03-04T15:26:58.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:57 smithi064 ceph-mon[30297]: purged_snaps scrub ok 2024-03-04T15:26:58.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:57 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:58.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:57 smithi064 ceph-mon[30297]: osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209] boot 2024-03-04T15:26:58.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:57 smithi064 ceph-mon[30297]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:26:58.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:57 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:58.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:57 smithi053 ceph-mon[25738]: purged_snaps scrub starts 2024-03-04T15:26:58.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:57 smithi053 ceph-mon[25738]: purged_snaps scrub ok 2024-03-04T15:26:58.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:57 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:58.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:57 smithi053 ceph-mon[25738]: osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209] boot 2024-03-04T15:26:58.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:57 smithi053 ceph-mon[25738]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:26:58.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:57 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:58.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:57 smithi160 ceph-mon[30717]: purged_snaps scrub starts 2024-03-04T15:26:58.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:57 smithi160 ceph-mon[30717]: purged_snaps scrub ok 2024-03-04T15:26:58.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:57 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:58.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:57 smithi160 ceph-mon[30717]: osd.0 [v2:172.21.15.53:6802/72549209,v1:172.21.15.53:6803/72549209] boot 2024-03-04T15:26:58.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:57 smithi160 ceph-mon[30717]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:26:58.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:57 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:26:58.389 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi064:/dev/nvme4n1 2024-03-04T15:26:58.625 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:26:59.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:59 smithi064 ceph-mon[30297]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:26:59.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:26:59 smithi064 ceph-mon[30297]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:26:59.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:59 smithi053 ceph-mon[25738]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:26:59.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:26:59 smithi053 ceph-mon[25738]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:26:59.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:59 smithi160 ceph-mon[30717]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:26:59.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:26:59 smithi160 ceph-mon[30717]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:00.068 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: Detected new or changed devices on smithi053 2024-03-04T15:27:00.068 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.068 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.068 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:00.068 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: Adjusting osd_memory_target on smithi053 to 17093M 2024-03-04T15:27:00.068 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.069 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:00.069 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:00.069 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.366 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: Detected new or changed devices on smithi053 2024-03-04T15:27:00.366 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.367 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.367 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:00.367 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: Adjusting osd_memory_target on smithi053 to 17093M 2024-03-04T15:27:00.367 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.367 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:00.367 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:00.367 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: Detected new or changed devices on smithi053 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: Adjusting osd_memory_target on smithi053 to 17093M 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:00.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:01.232 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:01 smithi064 ceph-mon[30297]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:01.232 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:01 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:27:01.232 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:01 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:27:01.232 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:01 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:01.232 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:01 smithi064 ceph-mon[30297]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:01.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:01 smithi053 ceph-mon[25738]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:01.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:27:01.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:27:01.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:01.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:01 smithi053 ceph-mon[25738]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:01.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:01 smithi160 ceph-mon[30717]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:01.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:01 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:27:01.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:01 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:27:01.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:01 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:01.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:01 smithi160 ceph-mon[30717]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:02.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:02 smithi160 ceph-mon[30717]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:03.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:02 smithi064 ceph-mon[30297]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:03.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:02 smithi053 ceph-mon[25738]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:03.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:03 smithi053 ceph-mon[25738]: from='client.? 172.21.15.64:0/230672760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]: dispatch 2024-03-04T15:27:03.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:03 smithi053 ceph-mon[25738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]: dispatch 2024-03-04T15:27:03.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:03 smithi053 ceph-mon[25738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]': finished 2024-03-04T15:27:03.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:03 smithi053 ceph-mon[25738]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:27:03.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:03 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:03.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:03 smithi160 ceph-mon[30717]: from='client.? 172.21.15.64:0/230672760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]: dispatch 2024-03-04T15:27:03.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:03 smithi160 ceph-mon[30717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]: dispatch 2024-03-04T15:27:03.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:03 smithi160 ceph-mon[30717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]': finished 2024-03-04T15:27:03.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:03 smithi160 ceph-mon[30717]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:27:03.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:03 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:04.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:03 smithi064 ceph-mon[30297]: from='client.? 172.21.15.64:0/230672760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]: dispatch 2024-03-04T15:27:04.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:03 smithi064 ceph-mon[30297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]: dispatch 2024-03-04T15:27:04.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:03 smithi064 ceph-mon[30297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394743b4-fef4-4238-a079-12ee548fafa0"}]': finished 2024-03-04T15:27:04.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:03 smithi064 ceph-mon[30297]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:27:04.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:03 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:04.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:04 smithi064 ceph-mon[30297]: from='client.? 172.21.15.64:0/1370788211' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:27:04.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:04 smithi064 ceph-mon[30297]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:04.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:04 smithi160 ceph-mon[30717]: from='client.? 172.21.15.64:0/1370788211' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:27:04.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:04 smithi160 ceph-mon[30717]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:05.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:04 smithi053 ceph-mon[25738]: from='client.? 172.21.15.64:0/1370788211' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:27:05.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:04 smithi053 ceph-mon[25738]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:06.929 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:06 smithi064 ceph-mon[30297]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:06 smithi160 ceph-mon[30717]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:07.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:06 smithi053 ceph-mon[25738]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:08.895 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:08 smithi064 ceph-mon[30297]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:08.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:08 smithi160 ceph-mon[30717]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:09.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:08 smithi053 ceph-mon[25738]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:10.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:10 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:27:10.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:10 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:10.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:10 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:27:10.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:10 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:10.521 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:10 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:27:10.521 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:10 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:11.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:11 smithi064 ceph-mon[30297]: Deploying daemon osd.1 on smithi064 2024-03-04T15:27:11.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:11 smithi064 ceph-mon[30297]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:11.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:11 smithi053 ceph-mon[25738]: Deploying daemon osd.1 on smithi064 2024-03-04T15:27:11.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:11 smithi053 ceph-mon[25738]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:11 smithi160 ceph-mon[30717]: Deploying daemon osd.1 on smithi064 2024-03-04T15:27:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:11 smithi160 ceph-mon[30717]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:12.836 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:12 smithi064 ceph-mon[30297]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:12.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:12 smithi160 ceph-mon[30717]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:13.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:12 smithi053 ceph-mon[25738]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:14.727 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:14 smithi064 ceph-mon[30297]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:14.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:14 smithi053 ceph-mon[25738]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:14.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:14 smithi160 ceph-mon[30717]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:16.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:16 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:16.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:16 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:16.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:16 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:16.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:16 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:16.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:16 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:16.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:16 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:16.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:16 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:16.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:16 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:16.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:16 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:17.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:17 smithi064 ceph-mon[30297]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:17.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:17 smithi053 ceph-mon[25738]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:17.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:17 smithi160 ceph-mon[30717]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:19.324 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 1 on host 'smithi064' 2024-03-04T15:27:19.521 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:27:19.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:27:19.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:19.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:19.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:19.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:19.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:27:19.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T15:27:19.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:20.026 DEBUG:teuthology.orchestra.run.smithi064:osd.1> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.1.service 2024-03-04T15:27:20.028 INFO:tasks.cephadm:Deploying osd.2 on smithi160 with /dev/nvme4n1... 2024-03-04T15:27:20.029 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-04T15:27:20.271 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:27:20.394 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:20.394 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:20 smithi160 ceph-mon[30717]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:27:20.394 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:20 smithi160 ceph-mon[30717]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:27:20.394 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:20.394 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:20 smithi160 ceph-mon[30717]: from='osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-04T15:27:20.394 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:20 smithi160 ceph-mon[30717]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-04T15:27:20.394 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:20.520 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:20.520 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-mon[30297]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:27:20.520 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-mon[30297]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:27:20.520 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:20.520 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-mon[30297]: from='osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-04T15:27:20.520 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-mon[30297]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-04T15:27:20.520 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:20.520 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1[34871]: 2024-03-04T15:27:20.314+0000 7fb351a0a700 -1 osd.1 0 waiting for initial osdmap 2024-03-04T15:27:20.521 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:20 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1[34871]: 2024-03-04T15:27:20.316+0000 7fb34d7fd700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:27:20.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:20.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:20 smithi053 ceph-mon[25738]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T15:27:20.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:20 smithi053 ceph-mon[25738]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T15:27:20.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:20.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:20 smithi053 ceph-mon[25738]: from='osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-04T15:27:20.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:20 smithi053 ceph-mon[25738]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-03-04T15:27:20.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:21.559 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:21 smithi064 ceph-mon[30297]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-04T15:27:21.560 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:21 smithi064 ceph-mon[30297]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:27:21.560 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:21.560 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:21.560 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:21 smithi064 ceph-mon[30297]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:21.580 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:21 smithi160 ceph-mon[30717]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-04T15:27:21.580 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:21 smithi160 ceph-mon[30717]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:27:21.580 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:21 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:21.580 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:21 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:21.580 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:21 smithi160 ceph-mon[30717]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:21.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:21 smithi053 ceph-mon[25738]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-03-04T15:27:21.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:21 smithi053 ceph-mon[25738]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T15:27:21.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:21 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:21.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:21 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:21.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:21 smithi053 ceph-mon[25738]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:27:22.099 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping: /dev/nvme4n1 2024-03-04T15:27:22.099 INFO:teuthology.orchestra.run.smithi160.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-04T15:27:22.099 INFO:teuthology.orchestra.run.smithi160.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-04T15:27:22.099 INFO:teuthology.orchestra.run.smithi160.stderr: stderr: 10+0 records in 2024-03-04T15:27:22.099 INFO:teuthology.orchestra.run.smithi160.stderr:10+0 records out 2024-03-04T15:27:22.099 INFO:teuthology.orchestra.run.smithi160.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126682 s, 828 MB/s 2024-03-04T15:27:22.099 INFO:teuthology.orchestra.run.smithi160.stderr:--> Zapping successful for: 2024-03-04T15:27:22.417 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:22 smithi160 ceph-mon[30717]: purged_snaps scrub starts 2024-03-04T15:27:22.418 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:22 smithi160 ceph-mon[30717]: purged_snaps scrub ok 2024-03-04T15:27:22.418 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:22 smithi160 ceph-mon[30717]: osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609] boot 2024-03-04T15:27:22.418 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:22 smithi160 ceph-mon[30717]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:27:22.418 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:22 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:22 smithi053 ceph-mon[25738]: purged_snaps scrub starts 2024-03-04T15:27:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:22 smithi053 ceph-mon[25738]: purged_snaps scrub ok 2024-03-04T15:27:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:22 smithi053 ceph-mon[25738]: osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609] boot 2024-03-04T15:27:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:22 smithi053 ceph-mon[25738]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:27:22.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:22 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:22.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi160:/dev/nvme4n1 2024-03-04T15:27:22.753 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:22 smithi064 ceph-mon[30297]: purged_snaps scrub starts 2024-03-04T15:27:22.754 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:22 smithi064 ceph-mon[30297]: purged_snaps scrub ok 2024-03-04T15:27:22.754 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:22 smithi064 ceph-mon[30297]: osd.1 [v2:172.21.15.64:6800/2416269609,v1:172.21.15.64:6801/2416269609] boot 2024-03-04T15:27:22.754 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:22 smithi064 ceph-mon[30297]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T15:27:22.754 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:22 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:27:22.956 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:23.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:23 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:23.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:23 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:23.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:23 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:24.385 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:24 smithi160 ceph-mon[30717]: Detected new or changed devices on smithi064 2024-03-04T15:27:24.385 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:24 smithi160 ceph-mon[30717]: Adjusting osd_memory_target on smithi064 to 17088M 2024-03-04T15:27:24.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:24 smithi053 ceph-mon[25738]: Detected new or changed devices on smithi064 2024-03-04T15:27:24.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:24 smithi053 ceph-mon[25738]: Adjusting osd_memory_target on smithi064 to 17088M 2024-03-04T15:27:24.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:24 smithi064 ceph-mon[30297]: Detected new or changed devices on smithi064 2024-03-04T15:27:24.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:24 smithi064 ceph-mon[30297]: Adjusting osd_memory_target on smithi064 to 17088M 2024-03-04T15:27:25.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:25 smithi053 ceph-mon[25738]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:25.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:27:25.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:27:25.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:25 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:25.712 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:25 smithi160 ceph-mon[30717]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:25.712 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:27:25.712 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:27:25.712 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:25 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:25.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:25 smithi064 ceph-mon[30297]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:25.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:27:25.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:27:25.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:25 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:26.464 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:26 smithi160 ceph-mon[30717]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:26.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:26 smithi053 ceph-mon[25738]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:26.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:26 smithi064 ceph-mon[30297]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi160:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:27.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:27 smithi053 ceph-mon[25738]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:27.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:27 smithi160 ceph-mon[30717]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:27.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:27 smithi064 ceph-mon[30297]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:28 smithi053 ceph-mon[25738]: from='client.? 172.21.15.160:0/4288707312' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]: dispatch 2024-03-04T15:27:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:28 smithi053 ceph-mon[25738]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]: dispatch 2024-03-04T15:27:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:28 smithi053 ceph-mon[25738]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]': finished 2024-03-04T15:27:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:28 smithi053 ceph-mon[25738]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:27:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:28 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:28 smithi160 ceph-mon[30717]: from='client.? 172.21.15.160:0/4288707312' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]: dispatch 2024-03-04T15:27:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:28 smithi160 ceph-mon[30717]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]: dispatch 2024-03-04T15:27:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:28 smithi160 ceph-mon[30717]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]': finished 2024-03-04T15:27:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:28 smithi160 ceph-mon[30717]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:27:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:28 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:28 smithi064 ceph-mon[30297]: from='client.? 172.21.15.160:0/4288707312' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]: dispatch 2024-03-04T15:27:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:28 smithi064 ceph-mon[30297]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]: dispatch 2024-03-04T15:27:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:28 smithi064 ceph-mon[30297]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d26aa6f4-50da-4bc4-807e-f84f084b889a"}]': finished 2024-03-04T15:27:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:28 smithi064 ceph-mon[30297]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T15:27:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:28 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:29.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:29 smithi053 ceph-mon[25738]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:29.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:29 smithi053 ceph-mon[25738]: from='client.? 172.21.15.160:0/75490884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:27:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:29 smithi160 ceph-mon[30717]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:29 smithi160 ceph-mon[30717]: from='client.? 172.21.15.160:0/75490884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:27:29.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:29 smithi064 ceph-mon[30297]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:29.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:29 smithi064 ceph-mon[30297]: from='client.? 172.21.15.160:0/75490884' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:27:30.958 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:30 smithi160 ceph-mon[30717]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:31.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:30 smithi064 ceph-mon[30297]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:31.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:30 smithi053 ceph-mon[25738]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:32.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:32 smithi160 ceph-mon[30717]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:33.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:32 smithi064 ceph-mon[30297]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:33.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:32 smithi053 ceph-mon[25738]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:34.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:34 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:27:34.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:34 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:34.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:34 smithi053 ceph-mon[25738]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:34.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:34 smithi053 ceph-mon[25738]: Deploying daemon osd.2 on smithi160 2024-03-04T15:27:34.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:34 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:27:34.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:34 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:34.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:34 smithi160 ceph-mon[30717]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:34.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:34 smithi160 ceph-mon[30717]: Deploying daemon osd.2 on smithi160 2024-03-04T15:27:35.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:34 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T15:27:35.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:34 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:35.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:34 smithi064 ceph-mon[30297]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:35.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:34 smithi064 ceph-mon[30297]: Deploying daemon osd.2 on smithi160 2024-03-04T15:27:36.871 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:36 smithi160 ceph-mon[30717]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:37.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:36 smithi064 ceph-mon[30297]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:37.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:36 smithi053 ceph-mon[25738]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:38.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:38 smithi160 ceph-mon[30717]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:39.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:38 smithi064 ceph-mon[30297]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:39.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:38 smithi053 ceph-mon[25738]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:41.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:41 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:41.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:41 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:41.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:41 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:41.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:41 smithi160 ceph-mon[30717]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:41.521 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:41 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:41.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:41 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:41.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:41 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:41.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:41 smithi064 ceph-mon[30297]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:41 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:41 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:41 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:41.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:41 smithi053 ceph-mon[25738]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:42.964 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:42 smithi160 ceph-mon[30717]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:43.023 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:42 smithi064 ceph-mon[30297]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:43.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:42 smithi053 ceph-mon[25738]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:43.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:43 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:43.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:43 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:43.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:43 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:43.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:43 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:43.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:43 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:43.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:43 smithi064 ceph-mon[30297]: from='osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:27:43.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:43 smithi064 ceph-mon[30297]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:27:43.890 INFO:teuthology.orchestra.run.smithi160.stdout:Created osd(s) 2 on host 'smithi160' 2024-03-04T15:27:43.891 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:43 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:43.891 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:43 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:43.891 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:43 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:43.891 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:43 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:43.891 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:43 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:43.891 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:43 smithi160 ceph-mon[30717]: from='osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:27:43.891 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:43 smithi160 ceph-mon[30717]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:27:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:43 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:43 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:43 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:43 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:43 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:43 smithi053 ceph-mon[25738]: from='osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:27:44.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:43 smithi053 ceph-mon[25738]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T15:27:44.479 DEBUG:teuthology.orchestra.run.smithi160:osd.2> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.2.service 2024-03-04T15:27:44.481 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-04T15:27:44.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T15:27:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:27:44.852 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: from='osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:44.853 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:44 smithi053 ceph-mon[25738]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:44.996 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:27:44.996 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:27:44.997 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:44.997 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: from='osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-03-04T15:27:44.997 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-03-04T15:27:44.997 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:44.997 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:44.997 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:44.997 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-mon[30717]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:44.997 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2[35980]: 2024-03-04T15:27:44.717+0000 7fc0d8a9b700 -1 osd.2 0 waiting for initial osdmap 2024-03-04T15:27:44.997 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:44 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2[35980]: 2024-03-04T15:27:44.719+0000 7fc0d488e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: from='osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]: dispatch 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:45.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:44 smithi064 ceph-mon[30297]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:45.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:45 smithi053 ceph-mon[25738]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-03-04T15:27:45.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:45 smithi053 ceph-mon[25738]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:27:45.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:45 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:45.824 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:45 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:45.985 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:45 smithi160 ceph-mon[30717]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-03-04T15:27:45.985 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:45 smithi160 ceph-mon[30717]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:27:45.985 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:45 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:45.985 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:45 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:46.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:45 smithi064 ceph-mon[30297]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi160", "root=default"]}]': finished 2024-03-04T15:27:46.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:45 smithi064 ceph-mon[30297]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T15:27:46.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:45 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:46.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:45 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:46.579 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:27:47.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: purged_snaps scrub starts 2024-03-04T15:27:47.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: purged_snaps scrub ok 2024-03-04T15:27:47.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:47.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447] boot 2024-03-04T15:27:47.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:27:47.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:47.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:47.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:46 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1461791719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:27:47.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: purged_snaps scrub starts 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: purged_snaps scrub ok 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447] boot 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:46 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1461791719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:27:47.272 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: purged_snaps scrub starts 2024-03-04T15:27:47.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: purged_snaps scrub ok 2024-03-04T15:27:47.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:47.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: osd.2 [v2:172.21.15.160:6800/914982447,v1:172.21.15.160:6801/914982447] boot 2024-03-04T15:27:47.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T15:27:47.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:27:47.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-04T15:27:47.273 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:46 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1461791719' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T15:27:47.298 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1709566065,"num_in_osds":3,"osd_in_since":1709566047,"num_remapped_pgs":0} 2024-03-04T15:27:47.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:27:47.537 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:27:47.924 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:47.925 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:47 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:48.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:48.216 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:47 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:48.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:47 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:48.515 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46589]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-04T15:27:48.515 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46589]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:48.515 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46589]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:48.515 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46589]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:48.814 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46592]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-04T15:27:48.814 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46592]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:48.815 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46592]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:48.815 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:27:48 smithi053 sudo[46592]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:49.062 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:48 smithi064 sudo[40043]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-04T15:27:49.062 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:48 smithi064 sudo[40043]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:49.063 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:48 smithi064 sudo[40043]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:49.063 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:48 smithi064 sudo[40043]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:49.063 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:48 smithi064 ceph-mon[30297]: Detected new or changed devices on smithi160 2024-03-04T15:27:49.063 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:48 smithi064 ceph-mon[30297]: Adjusting osd_memory_target on smithi160 to 17088M 2024-03-04T15:27:49.063 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:48 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:27:49.063 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:48 smithi064 ceph-mon[30297]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:27:49.063 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:48 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:49.063 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:48 smithi064 ceph-mon[30297]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:49.065 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:48 smithi053 ceph-mon[25738]: Detected new or changed devices on smithi160 2024-03-04T15:27:49.065 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:48 smithi053 ceph-mon[25738]: Adjusting osd_memory_target on smithi160 to 17088M 2024-03-04T15:27:49.065 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:48 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:27:49.065 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:48 smithi053 ceph-mon[25738]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:27:49.065 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:48 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:49.065 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:48 smithi053 ceph-mon[25738]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:49.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:48 smithi160 ceph-mon[30717]: Detected new or changed devices on smithi160 2024-03-04T15:27:49.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:48 smithi160 ceph-mon[30717]: Adjusting osd_memory_target on smithi160 to 17088M 2024-03-04T15:27:49.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:48 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T15:27:49.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:48 smithi160 ceph-mon[30717]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T15:27:49.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:48 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:49.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:48 smithi160 ceph-mon[30717]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:49.271 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:27:49.272 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":21,"fsid":"1fe97ada-da3b-11ee-95c6-87774f69a715","created":"2024-03-04T15:24:26.197024+0000","modified":"2024-03-04T15:27:48.899993+0000","last_up_change":"2024-03-04T15:27:45.714466+0000","last_in_change":"2024-03-04T15:27:27.473449+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:27:46.888483+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c0477ff-d13b-4394-902f-7800c9374c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6803","nonce":72549209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6805","nonce":72549209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6809","nonce":72549209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6807","nonce":72549209}]},"public_addr":"172.21.15.53:6803/72549209","cluster_addr":"172.21.15.53:6805/72549209","heartbeat_back_addr":"172.21.15.53:6809/72549209","heartbeat_front_addr":"172.21.15.53:6807/72549209","state":["exists","up"]},{"osd":1,"uuid":"394743b4-fef4-4238-a079-12ee548fafa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6801","nonce":2416269609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6803","nonce":2416269609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6807","nonce":2416269609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6805","nonce":2416269609}]},"public_addr":"172.21.15.64:6801/2416269609","cluster_addr":"172.21.15.64:6803/2416269609","heartbeat_back_addr":"172.21.15.64:6807/2416269609","heartbeat_front_addr":"172.21.15.64:6805/2416269609","state":["exists","up"]},{"osd":2,"uuid":"d26aa6f4-50da-4bc4-807e-f84f084b889a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6801","nonce":914982447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6803","nonce":914982447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6807","nonce":914982447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6805","nonce":914982447}]},"public_addr":"172.21.15.160:6801/914982447","cluster_addr":"172.21.15.160:6803/914982447","heartbeat_back_addr":"172.21.15.160:6807/914982447","heartbeat_front_addr":"172.21.15.160:6805/914982447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:26:55.169099+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:27:19.700452+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:27:44.223100+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6801/1575419801":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/2064139432":"2024-03-05T15:25:20.548668+0000","172.21.15.53:6800/1575419801":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/4221377914":"2024-03-05T15:25:20.548668+0000","172.21.15.53:6800/1308574311":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/2162042670":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/1374862187":"2024-03-05T15:24:53.250938+0000","172.21.15.53:6801/1308574311":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/2352019567":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/2570129014":"2024-03-05T15:24:53.250938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:27:49.521 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:49 smithi064 sudo[40046]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-04T15:27:49.522 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:49 smithi064 sudo[40046]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:49.522 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:49 smithi064 sudo[40046]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:49.522 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:27:49 smithi064 sudo[40046]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:49.581 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41505]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-04T15:27:49.582 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41505]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:49.582 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41505]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:49.582 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41505]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:49.888 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T15:27:46.888483+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T15:27:49.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-04T15:27:49.964 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41508]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-04T15:27:49.964 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41508]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:49.965 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41508]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:49.965 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:27:49 smithi160 sudo[41508]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:50.125 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:27:50.132 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:49 smithi053 sudo[46683]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:27:50.132 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:49 smithi053 sudo[46683]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:50.133 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:49 smithi053 sudo[46683]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:50.133 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:49 smithi053 sudo[46683]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:50.385 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 sudo[46744]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 sudo[46744]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 sudo[46744]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 sudo[46744]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 ceph-mon[25738]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1408838712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:50.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:50 smithi053 ceph-mon[25738]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:50.408 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 ceph-mon[30717]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:27:50.408 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:50.408 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1408838712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:27:50.408 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:50.408 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 ceph-mon[30717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:50.521 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 ceph-mon[30297]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T15:27:50.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:50.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1408838712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:27:50.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:50.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 ceph-mon[30297]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:50.659 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41511]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:27:50.659 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41511]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:50.659 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41511]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:50.659 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41511]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41514]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:27:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41514]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41514]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:50 smithi160 sudo[41514]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:51.202 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 sudo[40049]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T15:27:51.202 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 sudo[40049]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:51.202 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 sudo[40049]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:51.202 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:50 smithi064 sudo[40049]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:51.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:51.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-04T15:27:51.463 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.464 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:51 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:51.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:51.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:51.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.466 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:51 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 sudo[40052]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 sudo[40052]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 sudo[40052]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 sudo[40052]: pam_unix(sudo:session): session closed for user root 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-04T15:27:51.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:27:51.523 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:51 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:27:51.877 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-03-04T15:27:52.433 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi053 2024-03-04T15:27:52.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi053=foo.a' 2024-03-04T15:27:52.464 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:52 smithi160 ceph-mon[30717]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:52.464 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:52 smithi160 ceph-mon[30717]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:52.464 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:52 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:52.464 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:52 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/7948575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:27:52.521 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:52 smithi064 ceph-mon[30297]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:52.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:52 smithi064 ceph-mon[30297]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:52.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:52 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:52.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:52 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/7948575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:27:52.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:52 smithi053 ceph-mon[25738]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T15:27:52.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:52 smithi053 ceph-mon[25738]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T15:27:52.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:52 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:52.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:52 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/7948575' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T15:27:52.682 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:27:53.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:53 smithi160 ceph-mon[30717]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:53.521 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:53 smithi064 ceph-mon[30297]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:53.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:53 smithi053 ceph-mon[25738]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:54.434 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled rgw.foo.a update... 2024-03-04T15:27:55.042 DEBUG:teuthology.orchestra.run.smithi053:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@rgw.foo.a.service 2024-03-04T15:27:55.045 INFO:tasks.cephadm:Adding prometheus.a on smithi064 2024-03-04T15:27:55.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch apply prometheus '1;smithi064=a' 2024-03-04T15:27:55.314 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi053=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: Saving service rgw.foo.a spec with placement smithi053=foo.a;count:1 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:55 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:55.707 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi053=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:55.707 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: Saving service rgw.foo.a spec with placement smithi053=foo.a;count:1 2024-03-04T15:27:55.707 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.707 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:55.707 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:55.708 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:55.708 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:55.708 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.708 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T15:27:55.708 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T15:27:55.708 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.708 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:55 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:55.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi053=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:55.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: Saving service rgw.foo.a spec with placement smithi053=foo.a;count:1 2024-03-04T15:27:55.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:55.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:55.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:55.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:27:55.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T15:27:55.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T15:27:55.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:55.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:55 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:27:56.640 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:56 smithi160 ceph-mon[30717]: Deploying daemon rgw.foo.a on smithi053 2024-03-04T15:27:56.690 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:56 smithi053 ceph-mon[25738]: Deploying daemon rgw.foo.a on smithi053 2024-03-04T15:27:56.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:56 smithi064 ceph-mon[30297]: Deploying daemon rgw.foo.a on smithi053 2024-03-04T15:27:57.064 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled prometheus update... 2024-03-04T15:27:57.357 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:27:57 smithi053 systemd[1]: Starting Ceph rgw.foo.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:27:57.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:57 smithi053 ceph-mon[25738]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:57.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:57 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:57.636 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:27:57 smithi053 podman[47204]: 2024-03-04 15:27:57.355299587 +0000 UTC m=+0.092735830 container create 0062d4ed96bdba4a62b295c0aba669af15bb18b6dc0f4e6fd4d291f2b085aa3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T15:27:57.636 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:27:57 smithi053 podman[47204]: 2024-03-04 15:27:57.280131623 +0000 UTC m=+0.017567832 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:27:57.636 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:27:57 smithi053 podman[47204]: 2024-03-04 15:27:57.557672704 +0000 UTC m=+0.295108920 container init 0062d4ed96bdba4a62b295c0aba669af15bb18b6dc0f4e6fd4d291f2b085aa3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T15:27:57.636 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:27:57 smithi053 podman[47204]: 2024-03-04 15:27:57.563342448 +0000 UTC m=+0.300778650 container start 0062d4ed96bdba4a62b295c0aba669af15bb18b6dc0f4e6fd4d291f2b085aa3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-04T15:27:57.678 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:57 smithi160 ceph-mon[30717]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:57.678 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:57 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:57.705 DEBUG:teuthology.orchestra.run.smithi064:prometheus.a> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@prometheus.a.service 2024-03-04T15:27:57.707 INFO:tasks.cephadm:Adding node-exporter.a on smithi053 2024-03-04T15:27:57.708 INFO:tasks.cephadm:Adding node-exporter.b on smithi064 2024-03-04T15:27:57.708 INFO:tasks.cephadm:Adding node-exporter.c on smithi160 2024-03-04T15:27:57.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch apply node-exporter '3;smithi053=a;smithi064=b;smithi160=c' 2024-03-04T15:27:57.735 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:57 smithi064 ceph-mon[30297]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:57.735 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:57 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:57.941 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:27:58.135 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:27:57 smithi053 bash[47204]: 0062d4ed96bdba4a62b295c0aba669af15bb18b6dc0f4e6fd4d291f2b085aa3c 2024-03-04T15:27:58.136 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:27:57 smithi053 systemd[1]: Started Ceph rgw.foo.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: Saving service prometheus spec with placement smithi064=a;count:1 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:58 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:58.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:58.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: Saving service prometheus spec with placement smithi064=a;count:1 2024-03-04T15:27:58.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:58 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: Saving service prometheus spec with placement smithi064=a;count:1 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:27:58.861 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:58 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:27:59.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:59 smithi160 ceph-mon[30717]: Saving service rgw.foo.a spec with placement smithi053=foo.a;count:1 2024-03-04T15:27:59.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:59 smithi160 ceph-mon[30717]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:27:59.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:59 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/2629735164' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:27:59.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:27:59 smithi160 ceph-mon[30717]: pgmap v91: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:59.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:59 smithi064 ceph-mon[30297]: Saving service rgw.foo.a spec with placement smithi053=foo.a;count:1 2024-03-04T15:27:59.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:59 smithi064 ceph-mon[30297]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:27:59.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:59 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/2629735164' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:27:59.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:27:59 smithi064 ceph-mon[30297]: pgmap v91: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:27:59.807 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled node-exporter update... 2024-03-04T15:27:59.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:59 smithi053 ceph-mon[25738]: Saving service rgw.foo.a spec with placement smithi053=foo.a;count:1 2024-03-04T15:27:59.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:59 smithi053 ceph-mon[25738]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T15:27:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:59 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/2629735164' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T15:27:59.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:27:59 smithi053 ceph-mon[25738]: pgmap v91: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:00.407 DEBUG:teuthology.orchestra.run.smithi053:node-exporter.a> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.a.service 2024-03-04T15:28:00.410 DEBUG:teuthology.orchestra.run.smithi064:node-exporter.b> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.b.service 2024-03-04T15:28:00.412 DEBUG:teuthology.orchestra.run.smithi160:node-exporter.c> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.c.service 2024-03-04T15:28:00.415 INFO:tasks.cephadm:Adding alertmanager.a on smithi053 2024-03-04T15:28:00.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch apply alertmanager '1;smithi053=a' 2024-03-04T15:28:00.687 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/2629735164' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: Checking dashboard <-> RGW credentials 2024-03-04T15:28:00.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:00 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/2629735164' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: Checking dashboard <-> RGW credentials 2024-03-04T15:28:00.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:00 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/2629735164' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: osdmap e23: 3 total, 3 up, 3 in 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:28:00.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: Checking dashboard <-> RGW credentials 2024-03-04T15:28:00.887 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:00 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:01.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:01 smithi064 ceph-mon[30297]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi053=a;smithi064=b;smithi160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:01.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:01 smithi064 ceph-mon[30297]: Saving service node-exporter spec with placement smithi053=a;smithi064=b;smithi160=c;count:3 2024-03-04T15:28:01.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:01 smithi064 ceph-mon[30297]: osdmap e24: 3 total, 3 up, 3 in 2024-03-04T15:28:01.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:01 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:28:01.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:01 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:28:01.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:01 smithi064 ceph-mon[30297]: pgmap v94: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:01.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:01 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:01.822 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:01 smithi160 ceph-mon[30717]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi053=a;smithi064=b;smithi160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:01.822 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:01 smithi160 ceph-mon[30717]: Saving service node-exporter spec with placement smithi053=a;smithi064=b;smithi160=c;count:3 2024-03-04T15:28:01.822 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:01 smithi160 ceph-mon[30717]: osdmap e24: 3 total, 3 up, 3 in 2024-03-04T15:28:01.822 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:01 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:28:01.823 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:01 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:28:01.823 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:01 smithi160 ceph-mon[30717]: pgmap v94: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:01.823 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:01 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:01.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:01 smithi053 ceph-mon[25738]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi053=a;smithi064=b;smithi160=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:01 smithi053 ceph-mon[25738]: Saving service node-exporter spec with placement smithi053=a;smithi064=b;smithi160=c;count:3 2024-03-04T15:28:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:01 smithi053 ceph-mon[25738]: osdmap e24: 3 total, 3 up, 3 in 2024-03-04T15:28:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:01 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:28:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:01 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T15:28:01.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:01 smithi053 ceph-mon[25738]: pgmap v94: 65 pgs: 64 unknown, 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:01.887 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:01 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:02.571 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled alertmanager update... 2024-03-04T15:28:02.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:02 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:28:02.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:02 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:28:02.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:02 smithi064 ceph-mon[30297]: osdmap e25: 3 total, 3 up, 3 in 2024-03-04T15:28:02.876 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:02 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:28:02.876 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:02 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:28:02.876 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:02 smithi160 ceph-mon[30717]: osdmap e25: 3 total, 3 up, 3 in 2024-03-04T15:28:02.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:02 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:28:02.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:02 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T15:28:02.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:02 smithi053 ceph-mon[25738]: osdmap e25: 3 total, 3 up, 3 in 2024-03-04T15:28:03.204 DEBUG:teuthology.orchestra.run.smithi053:alertmanager.a> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@alertmanager.a.service 2024-03-04T15:28:03.207 INFO:tasks.cephadm:Adding grafana.a on smithi064 2024-03-04T15:28:03.207 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph orch apply grafana '1;smithi064=a' 2024-03-04T15:28:03.443 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:28:03.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: osdmap e26: 3 total, 3 up, 3 in 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: Saving service alertmanager spec with placement smithi053=a;count:1 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: pgmap v97: 97 pgs: 11 creating+peering, 52 unknown, 34 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:28:03.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:03 smithi053 ceph-mon[25738]: osdmap e27: 3 total, 3 up, 3 in 2024-03-04T15:28:03.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: osdmap e26: 3 total, 3 up, 3 in 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: Saving service alertmanager spec with placement smithi053=a;count:1 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: pgmap v97: 97 pgs: 11 creating+peering, 52 unknown, 34 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:28:03.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:03 smithi064 ceph-mon[30297]: osdmap e27: 3 total, 3 up, 3 in 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: osdmap e26: 3 total, 3 up, 3 in 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: Saving service alertmanager spec with placement smithi053=a;count:1 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: pgmap v97: 97 pgs: 11 creating+peering, 52 unknown, 34 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T15:28:03.805 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:03 smithi160 ceph-mon[30717]: osdmap e27: 3 total, 3 up, 3 in 2024-03-04T15:28:05.224 INFO:teuthology.orchestra.run.smithi160.stdout:Scheduled grafana update... 2024-03-04T15:28:05.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:05 smithi064 ceph-mon[30297]: osdmap e28: 3 total, 3 up, 3 in 2024-03-04T15:28:05.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:05 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:28:05.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:05 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:28:05.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:05 smithi064 ceph-mon[30297]: pgmap v100: 129 pgs: 22 creating+peering, 53 unknown, 54 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-04T15:28:05.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:05 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:05.814 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:05 smithi160 ceph-mon[30717]: osdmap e28: 3 total, 3 up, 3 in 2024-03-04T15:28:05.814 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:05 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:28:05.814 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:05 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:28:05.814 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:05 smithi160 ceph-mon[30717]: pgmap v100: 129 pgs: 22 creating+peering, 53 unknown, 54 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-04T15:28:05.814 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:05 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:05.840 DEBUG:teuthology.orchestra.run.smithi064:grafana.a> sudo journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@grafana.a.service 2024-03-04T15:28:05.843 INFO:tasks.cephadm:Setting up client nodes... 2024-03-04T15:28:05.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:28:05.872 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:05 smithi053 ceph-mon[25738]: osdmap e28: 3 total, 3 up, 3 in 2024-03-04T15:28:05.872 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:05 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:28:05.872 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:05 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T15:28:05.872 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:05 smithi053 ceph-mon[25738]: pgmap v100: 129 pgs: 22 creating+peering, 53 unknown, 54 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-04T15:28:05.873 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:05 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:06.096 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:06.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:06 smithi064 ceph-mon[30297]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:06.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:06 smithi064 ceph-mon[30297]: Saving service grafana spec with placement smithi064=a;count:1 2024-03-04T15:28:06.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:06 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:28:06.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:06 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:28:06.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:06 smithi064 ceph-mon[30297]: osdmap e29: 3 total, 3 up, 3 in 2024-03-04T15:28:06.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:06 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:28:06.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:06 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:28:06.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:06 smithi053 ceph-mon[25738]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:06.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:06 smithi053 ceph-mon[25738]: Saving service grafana spec with placement smithi064=a;count:1 2024-03-04T15:28:06.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:06 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:28:06.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:06 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:28:06.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:06 smithi053 ceph-mon[25738]: osdmap e29: 3 total, 3 up, 3 in 2024-03-04T15:28:06.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:06 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:28:06.886 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:06 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:28:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:06 smithi160 ceph-mon[30717]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:06 smithi160 ceph-mon[30717]: Saving service grafana spec with placement smithi064=a;count:1 2024-03-04T15:28:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:06 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:28:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:06 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T15:28:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:06 smithi160 ceph-mon[30717]: osdmap e29: 3 total, 3 up, 3 in 2024-03-04T15:28:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:06 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:28:06.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:06 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T15:28:07.534 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:07 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:28:07.534 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:07 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:28:07.534 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:07 smithi053 ceph-mon[25738]: osdmap e30: 3 total, 3 up, 3 in 2024-03-04T15:28:07.534 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:07 smithi053 ceph-mon[25738]: pgmap v103: 129 pgs: 11 creating+peering, 31 unknown, 87 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-04T15:28:07.534 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:07 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.534 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:07 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.535 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:07 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:07 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:28:07.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:07 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:28:07.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:07 smithi064 ceph-mon[30297]: osdmap e30: 3 total, 3 up, 3 in 2024-03-04T15:28:07.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:07 smithi064 ceph-mon[30297]: pgmap v103: 129 pgs: 11 creating+peering, 31 unknown, 87 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-04T15:28:07.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:07 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:07 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:07 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:07 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3807451217' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:28:07.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:07 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1210953626' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T15:28:07.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:07 smithi160 ceph-mon[30717]: osdmap e30: 3 total, 3 up, 3 in 2024-03-04T15:28:07.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:07 smithi160 ceph-mon[30717]: pgmap v103: 129 pgs: 11 creating+peering, 31 unknown, 87 active+clean; 578 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-04T15:28:07.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:07 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:07 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:07.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:07 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:08.093 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-03-04T15:28:08.093 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCI6OVlKodcBRAAIt4h3W81zG/EgY2O0H8A6Q== 2024-03-04T15:28:08.660 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:08 smithi053 ceph-mon[25738]: Deploying daemon prometheus.a on smithi064 2024-03-04T15:28:08.660 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:08 smithi053 ceph-mon[25738]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:28:08.660 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:08 smithi053 ceph-mon[25738]: Cluster is now healthy 2024-03-04T15:28:08.660 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:08 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1364445849' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:08.660 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:08 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1364445849' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:08.661 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:28:08.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-04T15:28:08.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T15:28:08.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:28:08.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:08 smithi064 ceph-mon[30297]: Deploying daemon prometheus.a on smithi064 2024-03-04T15:28:08.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:08 smithi064 ceph-mon[30297]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:28:08.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:08 smithi064 ceph-mon[30297]: Cluster is now healthy 2024-03-04T15:28:08.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:08 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1364445849' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:08.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:08 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1364445849' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:08.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:08 smithi160 ceph-mon[30717]: Deploying daemon prometheus.a on smithi064 2024-03-04T15:28:08.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:08 smithi160 ceph-mon[30717]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T15:28:08.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:08 smithi160 ceph-mon[30717]: Cluster is now healthy 2024-03-04T15:28:08.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:08 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1364445849' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:08.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:08 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1364445849' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:08.988 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:28:09.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:09 smithi064 ceph-mon[30297]: pgmap v104: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 4.9 KiB/s wr, 128 op/s 2024-03-04T15:28:09.885 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:09 smithi053 ceph-mon[25738]: pgmap v104: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 4.9 KiB/s wr, 128 op/s 2024-03-04T15:28:09.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:09 smithi160 ceph-mon[30717]: pgmap v104: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 54 KiB/s rd, 4.9 KiB/s wr, 128 op/s 2024-03-04T15:28:10.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:10 smithi064 ceph-mon[30297]: pgmap v105: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 4.1 KiB/s wr, 107 op/s 2024-03-04T15:28:10.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:10 smithi160 ceph-mon[30717]: pgmap v105: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 4.1 KiB/s wr, 107 op/s 2024-03-04T15:28:11.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:10 smithi053 ceph-mon[25738]: pgmap v105: 129 pgs: 129 active+clean; 581 KiB data, 90 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 4.1 KiB/s wr, 107 op/s 2024-03-04T15:28:11.421 INFO:teuthology.orchestra.run.smithi064.stdout:[client.1] 2024-03-04T15:28:11.421 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCL6OVlpXfAGBAAykhs38RLPjDtBKIy8CgGaw== 2024-03-04T15:28:11.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:11 smithi064 ceph-mon[30297]: from='client.? 172.21.15.64:0/4098283544' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:11.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:11 smithi064 ceph-mon[30297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:11.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:11 smithi064 ceph-mon[30297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:11.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:11 smithi160 ceph-mon[30717]: from='client.? 172.21.15.64:0/4098283544' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:11.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:11 smithi160 ceph-mon[30717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:11.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:11 smithi160 ceph-mon[30717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:12.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:11 smithi053 ceph-mon[25738]: from='client.? 172.21.15.64:0/4098283544' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:12.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:11 smithi053 ceph-mon[25738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:12.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:11 smithi053 ceph-mon[25738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:12.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:12 smithi160 ceph-mon[30717]: pgmap v106: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-04T15:28:13.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:12 smithi064 ceph-mon[30297]: pgmap v106: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-04T15:28:13.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:12 smithi053 ceph-mon[25738]: pgmap v106: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-04T15:28:13.484 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:28:13.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-04T15:28:13.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T15:28:13.570 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T15:28:13.816 INFO:teuthology.orchestra.run.smithi160.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.c/config 2024-03-04T15:28:14.902 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:14 smithi160 ceph-mon[30717]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 51 KiB/s rd, 4.0 KiB/s wr, 124 op/s 2024-03-04T15:28:14.994 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:14 smithi064 ceph-mon[30297]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 51 KiB/s rd, 4.0 KiB/s wr, 124 op/s 2024-03-04T15:28:15.135 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:14 smithi053 ceph-mon[25738]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 51 KiB/s rd, 4.0 KiB/s wr, 124 op/s 2024-03-04T15:28:15.700 INFO:teuthology.orchestra.run.smithi160.stdout:[client.2] 2024-03-04T15:28:15.700 INFO:teuthology.orchestra.run.smithi160.stdout: key = AQCP6OVlYj9hKRAAfbCbJIByuCPaMinAt/D97g== 2024-03-04T15:28:15.869 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:15 smithi064 ceph-mon[30297]: from='client.? 172.21.15.160:0/3009250164' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:15.869 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:15 smithi064 ceph-mon[30297]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:15.869 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:15 smithi064 ceph-mon[30297]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:15.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:15 smithi160 ceph-mon[30717]: from='client.? 172.21.15.160:0/3009250164' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:15.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:15 smithi160 ceph-mon[30717]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:15.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:15 smithi160 ceph-mon[30717]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:16.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:15 smithi053 ceph-mon[25738]: from='client.? 172.21.15.160:0/3009250164' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:16.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:15 smithi053 ceph-mon[25738]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T15:28:16.136 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:15 smithi053 ceph-mon[25738]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T15:28:16.327 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:28:16.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-04T15:28:16.327 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-04T15:28:16.368 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T15:28:16.368 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-04T15:28:16.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-04T15:28:16.624 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:17.012 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:16 smithi053 ceph-mon[25738]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 110 op/s 2024-03-04T15:28:17.021 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:16 smithi064 ceph-mon[30297]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 110 op/s 2024-03-04T15:28:17.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:16 smithi160 ceph-mon[30717]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 110 op/s 2024-03-04T15:28:18.272 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 systemd[1]: Starting Ceph prometheus.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:28:18.373 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:28:18.577 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3497600604' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:28:18.577 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 podman[40950]: 2024-03-04 15:28:18.222476471 +0000 UTC m=+0.018422941 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-04T15:28:18.577 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 podman[40950]: 2024-03-04 15:28:18.31919098 +0000 UTC m=+0.115137432 container create 71e0e568b872a616cf667687e5275b27b15937d5459626901208abe054fffd02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:28:18.577 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 podman[40950]: 2024-03-04 15:28:18.546059549 +0000 UTC m=+0.342006013 container init 71e0e568b872a616cf667687e5275b27b15937d5459626901208abe054fffd02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:28:18.577 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 podman[40950]: 2024-03-04 15:28:18.551928201 +0000 UTC m=+0.347874653 container start 71e0e568b872a616cf667687e5275b27b15937d5459626901208abe054fffd02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:28:18.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:18 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3497600604' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:28:18.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:18 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3497600604' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T15:28:18.983 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":3878037507},{"type":"v1","addr":"172.21.15.53:6801","nonce":3878037507}]},"active_addr":"172.21.15.53:6801/3878037507","active_change":"2024-03-04T15:25:20.548776+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.53:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":333352897}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1447371624}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":4114181096}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":1639874968}]}]} 2024-03-04T15:28:18.989 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-04T15:28:18.989 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T15:28:18.989 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.575Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.575Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.575Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi064 (none))" 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.575Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.575Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.577Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.577Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.578Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-04T15:28:19.022 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.579Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.580Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.580Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.305µs 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.580Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.580Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.580Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.042µs wal_replay_duration=179.118µs wbl_replay_duration=132ns total_replay_duration=235.98µs 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.581Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.581Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.581Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.602Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.126901ms db_storage=688ns remote_storage=1.004µs web_handler=209ns query_engine=516ns scrape=404.29µs scrape_sd=90.088µs notify=918ns notify_sd=2.521µs rules=20.375538ms tracing=6.254µs 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.602Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-04T15:28:19.023 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:18 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:28:18.602Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-04T15:28:19.232 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:19.607 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:19 smithi053 ceph-mon[25738]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 93 op/s 2024-03-04T15:28:19.607 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:19.607 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:19 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:19.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:19 smithi160 ceph-mon[30717]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 93 op/s 2024-03-04T15:28:19.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:19.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:19 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:19.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:19 smithi064 ceph-mon[30297]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 38 KiB/s rd, 2.9 KiB/s wr, 93 op/s 2024-03-04T15:28:19.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:19.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:19 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:19.772 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:19 smithi064 bash[40950]: 71e0e568b872a616cf667687e5275b27b15937d5459626901208abe054fffd02 2024-03-04T15:28:19.772 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:28:19 smithi064 systemd[1]: Started Ceph prometheus.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:28:20.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:20.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:20 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T15:28:20.628 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:20 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ignoring --setuser ceph since I am not root 2024-03-04T15:28:20.628 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:20 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ignoring --setgroup ceph since I am not root 2024-03-04T15:28:20.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:20.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:20 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T15:28:20.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:20 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: ignoring --setuser ceph since I am not root 2024-03-04T15:28:20.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:20 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: ignoring --setgroup ceph since I am not root 2024-03-04T15:28:20.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:20 smithi160 ceph-mgr[32823]: -- 172.21.15.160:0/84323836 <== mon.1 v2:172.21.15.160:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bc4059c5a0 con 0x55bc40571000 2024-03-04T15:28:20.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:20 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' 2024-03-04T15:28:20.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:20 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T15:28:20.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:20 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: ignoring --setuser ceph since I am not root 2024-03-04T15:28:20.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:20 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: ignoring --setgroup ceph since I am not root 2024-03-04T15:28:20.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:20 smithi064 ceph-mgr[31785]: -- 172.21.15.64:0/2634597342 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555bc5f405a0 con 0x555bc5f14800 2024-03-04T15:28:21.102 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:28:21.102 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":30,"fsid":"1fe97ada-da3b-11ee-95c6-87774f69a715","created":"2024-03-04T15:24:26.197024+0000","modified":"2024-03-04T15:28:06.486353+0000","last_up_change":"2024-03-04T15:27:45.714466+0000","last_in_change":"2024-03-04T15:27:27.473449+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:27:46.888483+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-04T15:27:57.849417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-04T15:27:59.768560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-04T15:28:01.727766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-04T15:28:03.794768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c0477ff-d13b-4394-902f-7800c9374c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6803","nonce":72549209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6805","nonce":72549209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6809","nonce":72549209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6807","nonce":72549209}]},"public_addr":"172.21.15.53:6803/72549209","cluster_addr":"172.21.15.53:6805/72549209","heartbeat_back_addr":"172.21.15.53:6809/72549209","heartbeat_front_addr":"172.21.15.53:6807/72549209","state":["exists","up"]},{"osd":1,"uuid":"394743b4-fef4-4238-a079-12ee548fafa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6801","nonce":2416269609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6803","nonce":2416269609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6807","nonce":2416269609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6805","nonce":2416269609}]},"public_addr":"172.21.15.64:6801/2416269609","cluster_addr":"172.21.15.64:6803/2416269609","heartbeat_back_addr":"172.21.15.64:6807/2416269609","heartbeat_front_addr":"172.21.15.64:6805/2416269609","state":["exists","up"]},{"osd":2,"uuid":"d26aa6f4-50da-4bc4-807e-f84f084b889a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6801","nonce":914982447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6803","nonce":914982447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6807","nonce":914982447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6805","nonce":914982447}]},"public_addr":"172.21.15.160:6801/914982447","cluster_addr":"172.21.15.160:6803/914982447","heartbeat_back_addr":"172.21.15.160:6807/914982447","heartbeat_front_addr":"172.21.15.160:6805/914982447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:26:55.169099+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:27:19.700452+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:27:44.223100+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6801/1575419801":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/2064139432":"2024-03-05T15:25:20.548668+0000","172.21.15.53:6800/1575419801":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/4221377914":"2024-03-05T15:25:20.548668+0000","172.21.15.53:6800/1308574311":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/2162042670":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/1374862187":"2024-03-05T15:24:53.250938+0000","172.21.15.53:6801/1308574311":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/2352019567":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/2570129014":"2024-03-05T15:24:53.250938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:28:21.157 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:21 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:21.054+0000 7f943d5b7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:28:21.408 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:21 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:21.155+0000 7f943d5b7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:28:21.465 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:21.025+0000 7fd7ee460200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:28:21.465 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:21.118+0000 7fd7ee460200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:28:21.465 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:21 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:21.034+0000 7f6056b6c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:28:21.465 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:21 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:21.129+0000 7f6056b6c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:28:21.659 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:21 smithi053 ceph-mon[25738]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T15:28:21.659 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:21 smithi053 ceph-mon[25738]: mgrmap e17: y(active, since 2m), standbys: z, x 2024-03-04T15:28:21.659 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:21 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/2956819342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:28:21.686 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T15:28:21.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T15:28:21.759 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-mon[30717]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T15:28:21.759 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-mon[30717]: mgrmap e17: y(active, since 2m), standbys: z, x 2024-03-04T15:28:21.759 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/2956819342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:28:21.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:21 smithi064 ceph-mon[30297]: from='mgr.14152 172.21.15.53:0/1541229008' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T15:28:21.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:21 smithi064 ceph-mon[30297]: mgrmap e17: y(active, since 2m), standbys: z, x 2024-03-04T15:28:21.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:21 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/2956819342' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:28:21.932 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:21.933 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:21 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:21.825+0000 7f943d5b7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:28:21.933 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:21 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:21.930+0000 7f943d5b7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:28:22.090 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:21.756+0000 7fd7ee460200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:28:22.091 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:21.850+0000 7fd7ee460200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:28:22.091 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:21 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:21.992+0000 7fd7ee460200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:28:22.091 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:22 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:22.088+0000 7fd7ee460200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:28:22.119 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:21 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:21.783+0000 7f6056b6c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:28:22.119 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:21 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:21.877+0000 7f6056b6c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:28:22.120 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.020+0000 7f6056b6c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:28:22.120 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.117+0000 7f6056b6c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:28:22.187 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:22.086+0000 7f943d5b7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:28:22.187 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:22.185+0000 7f943d5b7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:28:22.447 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:22 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:22.172+0000 7fd7ee460200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:28:22.447 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:22 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:22.445+0000 7fd7ee460200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:28:22.470 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.201+0000 7f6056b6c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:28:22.550 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:22.275+0000 7f943d5b7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:28:22.550 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:22.548+0000 7f943d5b7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:28:22.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:22 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:22.555+0000 7fd7ee460200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:28:22.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:22 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:22.643+0000 7fd7ee460200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:28:22.766 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.468+0000 7f6056b6c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:28:22.766 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.580+0000 7f6056b6c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:28:22.766 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.667+0000 7f6056b6c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:28:22.766 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.764+0000 7f6056b6c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:28:22.853 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:22.661+0000 7f943d5b7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:28:22.853 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:22.753+0000 7f943d5b7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:28:23.021 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:22.940+0000 7f6056b6c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:28:23.135 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:22.851+0000 7f943d5b7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:28:23.135 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:23 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:23.031+0000 7f943d5b7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:28:23.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:22 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:22.740+0000 7fd7ee460200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:28:23.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:22 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:22.917+0000 7fd7ee460200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:28:23.773 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:28:23.773 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":30,"fsid":"1fe97ada-da3b-11ee-95c6-87774f69a715","created":"2024-03-04T15:24:26.197024+0000","modified":"2024-03-04T15:28:06.486353+0000","last_up_change":"2024-03-04T15:27:45.714466+0000","last_in_change":"2024-03-04T15:27:27.473449+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T15:27:46.888483+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-04T15:27:57.849417+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-04T15:27:59.768560+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-04T15:28:01.727766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-04T15:28:03.794768+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c0477ff-d13b-4394-902f-7800c9374c58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6803","nonce":72549209}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6805","nonce":72549209}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6809","nonce":72549209}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":72549209},{"type":"v1","addr":"172.21.15.53:6807","nonce":72549209}]},"public_addr":"172.21.15.53:6803/72549209","cluster_addr":"172.21.15.53:6805/72549209","heartbeat_back_addr":"172.21.15.53:6809/72549209","heartbeat_front_addr":"172.21.15.53:6807/72549209","state":["exists","up"]},{"osd":1,"uuid":"394743b4-fef4-4238-a079-12ee548fafa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6801","nonce":2416269609}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6803","nonce":2416269609}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6807","nonce":2416269609}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":2416269609},{"type":"v1","addr":"172.21.15.64:6805","nonce":2416269609}]},"public_addr":"172.21.15.64:6801/2416269609","cluster_addr":"172.21.15.64:6803/2416269609","heartbeat_back_addr":"172.21.15.64:6807/2416269609","heartbeat_front_addr":"172.21.15.64:6805/2416269609","state":["exists","up"]},{"osd":2,"uuid":"d26aa6f4-50da-4bc4-807e-f84f084b889a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6800","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6801","nonce":914982447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6802","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6803","nonce":914982447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6806","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6807","nonce":914982447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.160:6804","nonce":914982447},{"type":"v1","addr":"172.21.15.160:6805","nonce":914982447}]},"public_addr":"172.21.15.160:6801/914982447","cluster_addr":"172.21.15.160:6803/914982447","heartbeat_back_addr":"172.21.15.160:6807/914982447","heartbeat_front_addr":"172.21.15.160:6805/914982447","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:26:55.169099+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:27:19.700452+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T15:27:44.223100+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6801/1575419801":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/2064139432":"2024-03-05T15:25:20.548668+0000","172.21.15.53:6800/1575419801":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/4221377914":"2024-03-05T15:25:20.548668+0000","172.21.15.53:6800/1308574311":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/2162042670":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/1374862187":"2024-03-05T15:24:53.250938+0000","172.21.15.53:6801/1308574311":"2024-03-05T15:24:53.250938+0000","172.21.15.53:0/2352019567":"2024-03-05T15:25:20.548668+0000","172.21.15.53:0/2570129014":"2024-03-05T15:24:53.250938+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T15:28:24.047 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:23 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3293891579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:28:24.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:23 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3293891579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:28:24.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:23 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:23.888+0000 7fd7ee460200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:28:24.215 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:24 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:24.020+0000 7fd7ee460200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:28:24.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:23 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3293891579' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T15:28:24.272 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:23 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:23.922+0000 7f6056b6c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:28:24.272 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:24 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:24.055+0000 7f6056b6c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:28:24.321 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:24 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:24.045+0000 7f943d5b7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:28:24.321 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:24 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:24.181+0000 7f943d5b7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:28:24.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T15:28:24.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T15:28:24.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T15:28:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:24.635 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:24.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:24 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:24.538+0000 7f943d5b7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:28:24.639 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:24.650 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:24 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:24.355+0000 7fd7ee460200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:28:24.650 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:24 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:24.448+0000 7fd7ee460200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:28:24.650 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:24 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:24.647+0000 7fd7ee460200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:28:24.687 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:24 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:24.395+0000 7f6056b6c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:28:24.687 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:24 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:24.486+0000 7f6056b6c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:28:24.687 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:24 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:24.685+0000 7f6056b6c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:28:24.965 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:24 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:24.835+0000 7fd7ee460200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:28:25.007 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:24 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:24.674+0000 7f943d5b7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:28:25.007 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:24 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:24.879+0000 7f943d5b7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:28:25.021 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:24 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:24.873+0000 7f6056b6c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:28:25.318 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:25 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:25.074+0000 7f943d5b7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:28:25.387 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:25 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:25.123+0000 7fd7ee460200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:28:25.387 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:25 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:25.385+0000 7fd7ee460200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:28:25.427 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:25 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:25.164+0000 7f6056b6c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:28:25.427 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:25 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:25.425+0000 7f6056b6c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:28:25.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:25 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:25.382+0000 7f943d5b7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:28:25.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:25 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:25.622+0000 7fd7ee460200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:28:25.771 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:25 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:25.667+0000 7f6056b6c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:28:25.914 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:25 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:25.647+0000 7f943d5b7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:28:26.164 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:25 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:25.912+0000 7f943d5b7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:28:26.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:26 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:26.429+0000 7fd7ee460200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:28:26.771 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:26 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:26.493+0000 7f6056b6c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:28:26.851 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:26 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:26.752+0000 7f943d5b7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:28:27.322 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:27.003+0000 7fd7ee460200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:28:27.322 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:27.102+0000 7fd7ee460200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:28:27.322 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:27.319+0000 7fd7ee460200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:28:27.394 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:27.072+0000 7f6056b6c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:28:27.394 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:27.172+0000 7f6056b6c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:28:27.394 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:27.391+0000 7f6056b6c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:28:27.490 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:27.350+0000 7f943d5b7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:28:27.490 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:27.459+0000 7f943d5b7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:28:27.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-mon[25738]: Standby manager daemon z restarted 2024-03-04T15:28:27.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-mon[25738]: Standby manager daemon z started 2024-03-04T15:28:27.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:28:27.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:28:27.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:28:27.491 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-mon[30717]: Standby manager daemon z restarted 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-mon[30717]: Standby manager daemon z started 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: [04/Mar/2024:15:28:27] ENGINE Bus STARTING 2024-03-04T15:28:27.715 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: CherryPy Checker: 2024-03-04T15:28:27.716 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: The Application mounted at '' has an empty config. 2024-03-04T15:28:27.716 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: 2024-03-04T15:28:27.716 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: [04/Mar/2024:15:28:27] ENGINE Serving on http://:::9283 2024-03-04T15:28:27.716 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:28:27 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z[32800]: [04/Mar/2024:15:28:27] ENGINE Bus STARTED 2024-03-04T15:28:27.743 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:27.694+0000 7f943d5b7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:28:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-mon[30297]: Standby manager daemon z restarted 2024-03-04T15:28:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-mon[30297]: Standby manager daemon z started 2024-03-04T15:28:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-04T15:28:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:28:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-04T15:28:27.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.160:0/1532117751' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:28:27.772 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: [04/Mar/2024:15:28:27] ENGINE Bus STARTING 2024-03-04T15:28:27.773 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: CherryPy Checker: 2024-03-04T15:28:27.773 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: The Application mounted at '' has an empty config. 2024-03-04T15:28:27.773 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: 2024-03-04T15:28:27.773 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: [04/Mar/2024:15:28:27] ENGINE Serving on http://:::9283 2024-03-04T15:28:27.773 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:28:27 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x[31761]: [04/Mar/2024:15:28:27] ENGINE Bus STARTED 2024-03-04T15:28:28.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:27] ENGINE Bus STARTING 2024-03-04T15:28:28.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: CherryPy Checker: 2024-03-04T15:28:28.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: The Application mounted at '' has an empty config. 2024-03-04T15:28:28.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:28:28.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:27] ENGINE Serving on http://:::9283 2024-03-04T15:28:28.136 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:27 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:27] ENGINE Bus STARTED 2024-03-04T15:28:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-04T15:28:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: Standby manager daemon x restarted 2024-03-04T15:28:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: Standby manager daemon x started 2024-03-04T15:28:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:28:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:28:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:28:28.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: Active manager daemon y restarted 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: Activating manager daemon y 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: osdmap e31: 3 total, 3 up, 3 in 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: mgrmap e19: y(active, starting, since 0.0108044s), standbys: x, z 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:28:28.637 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: Manager daemon y is now available 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:28:28.638 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:28 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: Standby manager daemon x restarted 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: Standby manager daemon x started 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: Active manager daemon y restarted 2024-03-04T15:28:28.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: Activating manager daemon y 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: osdmap e31: 3 total, 3 up, 3 in 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: mgrmap e19: y(active, starting, since 0.0108044s), standbys: x, z 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:28:28.716 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: Manager daemon y is now available 2024-03-04T15:28:28.717 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:28:28.717 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:28:28.717 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:28.717 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:28.717 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:28:28.717 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:28 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: Standby manager daemon x restarted 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: Standby manager daemon x started 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.? 172.21.15.64:0/2409494506' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:28:28.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: Active manager daemon y restarted 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: Activating manager daemon y 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: osdmap e31: 3 total, 3 up, 3 in 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: mgrmap e19: y(active, starting, since 0.0108044s), standbys: x, z 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:28:28.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:28:28.774 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: Manager daemon y is now available 2024-03-04T15:28:28.774 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:28:28.774 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:28:28.774 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:28.774 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:28.774 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:28:28.774 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:28 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:28:28.975 INFO:teuthology.orchestra.run.smithi053.stdout:77309411337 2024-03-04T15:28:28.975 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T15:28:29.612 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:29 smithi053 ceph-mon[25738]: [04/Mar/2024:15:28:28] ENGINE Bus STARTING 2024-03-04T15:28:29.612 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:29 smithi053 ceph-mon[25738]: mgrmap e20: y(active, since 1.01405s), standbys: x, z 2024-03-04T15:28:29.714 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:29 smithi160 ceph-mon[30717]: [04/Mar/2024:15:28:28] ENGINE Bus STARTING 2024-03-04T15:28:29.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:29 smithi160 ceph-mon[30717]: mgrmap e20: y(active, since 1.01405s), standbys: x, z 2024-03-04T15:28:29.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:29 smithi064 ceph-mon[30297]: [04/Mar/2024:15:28:28] ENGINE Bus STARTING 2024-03-04T15:28:29.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:29 smithi064 ceph-mon[30297]: mgrmap e20: y(active, since 1.01405s), standbys: x, z 2024-03-04T15:28:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:29.927 INFO:teuthology.orchestra.run.smithi053.stdout:34359738387 2024-03-04T15:28:29.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: [04/Mar/2024:15:28:28] ENGINE Serving on https://172.21.15.53:7150 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: [04/Mar/2024:15:28:28] ENGINE Serving on http://172.21.15.53:8765 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: [04/Mar/2024:15:28:28] ENGINE Bus STARTED 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:30 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.728 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: [04/Mar/2024:15:28:28] ENGINE Serving on https://172.21.15.53:7150 2024-03-04T15:28:30.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: [04/Mar/2024:15:28:28] ENGINE Serving on http://172.21.15.53:8765 2024-03-04T15:28:30.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: [04/Mar/2024:15:28:28] ENGINE Bus STARTED 2024-03-04T15:28:30.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:30.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.729 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:30 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: [04/Mar/2024:15:28:28] ENGINE Serving on https://172.21.15.53:7150 2024-03-04T15:28:30.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: [04/Mar/2024:15:28:28] ENGINE Serving on http://172.21.15.53:8765 2024-03-04T15:28:30.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: [04/Mar/2024:15:28:28] ENGINE Bus STARTED 2024-03-04T15:28:30.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:30.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:30.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:30 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.504 INFO:teuthology.orchestra.run.smithi053.stdout:55834574862 2024-03-04T15:28:31.505 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:28:31.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:31 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:31.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-04T15:28:31.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:28:31.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:31 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:28:31.845 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:31 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:32.466 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:32 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ::ffff:172.21.15.64 - - [04/Mar/2024:15:28:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:28:32.758 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:32 smithi053 ceph-mon[25738]: Adjusting osd_memory_target on smithi064 to 16064M 2024-03-04T15:28:32.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:32 smithi064 ceph-mon[30297]: Adjusting osd_memory_target on smithi064 to 16064M 2024-03-04T15:28:32.854 INFO:teuthology.orchestra.run.smithi053.stdout:77309411338 2024-03-04T15:28:32.965 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:32.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:32 smithi160 ceph-mon[30717]: Adjusting osd_memory_target on smithi064 to 16064M 2024-03-04T15:28:33.489 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:33 smithi053 ceph-mon[25738]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-04T15:28:33.489 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:33 smithi053 ceph-mon[25738]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-04T15:28:33.489 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:33 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/2338784938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T15:28:33.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411337 got 77309411338 for osd.2 2024-03-04T15:28:33.514 DEBUG:teuthology.parallel:result is None 2024-03-04T15:28:33.529 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:33.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:33 smithi064 ceph-mon[30297]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-04T15:28:33.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:33 smithi064 ceph-mon[30297]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-04T15:28:33.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:33 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/2338784938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T15:28:33.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:33 smithi160 ceph-mon[30717]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-03-04T15:28:33.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:33 smithi160 ceph-mon[30717]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-03-04T15:28:33.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:33 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/2338784938' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T15:28:35.734 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:35 smithi053 ceph-mon[25738]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-04T15:28:35.735 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:35 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:35.735 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:35 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:35.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:35 smithi064 ceph-mon[30297]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-04T15:28:35.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:35 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:35.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:35 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:35.876 INFO:teuthology.orchestra.run.smithi053.stdout:34359738389 2024-03-04T15:28:35.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:35 smithi160 ceph-mon[30717]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-04T15:28:35.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:35 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:35.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:35 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:36.530 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:36 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/221450920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T15:28:36.531 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738387 got 34359738389 for osd.0 2024-03-04T15:28:36.531 DEBUG:teuthology.parallel:result is None 2024-03-04T15:28:36.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:36 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/221450920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T15:28:36.964 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:36 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/221450920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T15:28:37.368 INFO:teuthology.orchestra.run.smithi053.stdout:55834574863 2024-03-04T15:28:37.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T15:28:37.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:28:37.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:28:37.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:28:37.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:37 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/1623111719' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T15:28:37.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T15:28:37.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:28:37.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:28:37.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:28:37.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:37 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/1623111719' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:28:37.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:37 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/1623111719' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T15:28:38.008 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574862 got 55834574863 for osd.1 2024-03-04T15:28:38.008 DEBUG:teuthology.parallel:result is None 2024-03-04T15:28:38.008 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T15:28:38.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T15:28:38.255 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:38.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Adjusting osd_memory_target on smithi053 to 16069M 2024-03-04T15:28:38.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-04T15:28:38.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:28:38.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:28:38.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.627 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.628 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:38 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Adjusting osd_memory_target on smithi053 to 16069M 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:38 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Adjusting osd_memory_target on smithi053 to 16069M 2024-03-04T15:28:38.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi053:/etc/ceph/ceph.conf 2024-03-04T15:28:38.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi064:/etc/ceph/ceph.conf 2024-03-04T15:28:38.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi160:/etc/ceph/ceph.conf 2024-03-04T15:28:38.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi160:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: Updating smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.966 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:38.967 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:38 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:39.740 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:39 smithi053 ceph-mon[25738]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T15:28:39.740 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:39 smithi053 ceph-mon[25738]: Deploying daemon alertmanager.a on smithi053 2024-03-04T15:28:39.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:39 smithi064 ceph-mon[30297]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T15:28:39.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:39 smithi064 ceph-mon[30297]: Deploying daemon alertmanager.a on smithi053 2024-03-04T15:28:39.964 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:39 smithi160 ceph-mon[30717]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-04T15:28:39.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:39 smithi160 ceph-mon[30717]: Deploying daemon alertmanager.a on smithi053 2024-03-04T15:28:40.251 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:28:40.254 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-03-04T15:28:41.455 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-04T15:28:39.726368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":100780,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":281122388,"statfs":{"total":287972524032,"available":287869325312,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.001377"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716024+0000","last_change":"2024-03-04T15:28:05.497122+0000","last_active":"2024-03-04T15:28:27.716024+0000","last_peered":"2024-03-04T15:28:27.716024+0000","last_clean":"2024-03-04T15:28:27.716024+0000","last_became_active":"2024-03-04T15:28:05.496672+0000","last_became_peered":"2024-03-04T15:28:05.496672+0000","last_unstale":"2024-03-04T15:28:27.716024+0000","last_undegraded":"2024-03-04T15:28:27.716024+0000","last_fullsized":"2024-03-04T15:28:27.716024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:46.473477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717800+0000","last_change":"2024-03-04T15:28:03.479478+0000","last_active":"2024-03-04T15:28:27.717800+0000","last_peered":"2024-03-04T15:28:27.717800+0000","last_clean":"2024-03-04T15:28:27.717800+0000","last_became_active":"2024-03-04T15:28:03.478632+0000","last_became_peered":"2024-03-04T15:28:03.478632+0000","last_unstale":"2024-03-04T15:28:27.717800+0000","last_undegraded":"2024-03-04T15:28:27.717800+0000","last_fullsized":"2024-03-04T15:28:27.717800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:28:30.174967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718190+0000","last_change":"2024-03-04T15:27:59.464910+0000","last_active":"2024-03-04T15:28:27.718190+0000","last_peered":"2024-03-04T15:28:27.718190+0000","last_clean":"2024-03-04T15:28:27.718190+0000","last_became_active":"2024-03-04T15:27:59.464339+0000","last_became_peered":"2024-03-04T15:27:59.464339+0000","last_unstale":"2024-03-04T15:28:27.718190+0000","last_undegraded":"2024-03-04T15:28:27.718190+0000","last_fullsized":"2024-03-04T15:28:27.718190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:39.262425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716825+0000","last_change":"2024-03-04T15:28:01.478830+0000","last_active":"2024-03-04T15:28:27.716825+0000","last_peered":"2024-03-04T15:28:27.716825+0000","last_clean":"2024-03-04T15:28:27.716825+0000","last_became_active":"2024-03-04T15:28:01.478400+0000","last_became_peered":"2024-03-04T15:28:01.478400+0000","last_unstale":"2024-03-04T15:28:27.716825+0000","last_undegraded":"2024-03-04T15:28:27.716825+0000","last_fullsized":"2024-03-04T15:28:27.716825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:20:31.901452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713982+0000","last_change":"2024-03-04T15:28:05.500883+0000","last_active":"2024-03-04T15:28:27.713982+0000","last_peered":"2024-03-04T15:28:27.713982+0000","last_clean":"2024-03-04T15:28:27.713982+0000","last_became_active":"2024-03-04T15:28:05.500360+0000","last_became_peered":"2024-03-04T15:28:05.500360+0000","last_unstale":"2024-03-04T15:28:27.713982+0000","last_undegraded":"2024-03-04T15:28:27.713982+0000","last_fullsized":"2024-03-04T15:28:27.713982+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:22.452537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715852+0000","last_change":"2024-03-04T15:28:03.485974+0000","last_active":"2024-03-04T15:28:27.715852+0000","last_peered":"2024-03-04T15:28:27.715852+0000","last_clean":"2024-03-04T15:28:27.715852+0000","last_became_active":"2024-03-04T15:28:03.485631+0000","last_became_peered":"2024-03-04T15:28:03.485631+0000","last_unstale":"2024-03-04T15:28:27.715852+0000","last_undegraded":"2024-03-04T15:28:27.715852+0000","last_fullsized":"2024-03-04T15:28:27.715852+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:57:10.172676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717224+0000","last_change":"2024-03-04T15:27:59.465221+0000","last_active":"2024-03-04T15:28:27.717224+0000","last_peered":"2024-03-04T15:28:27.717224+0000","last_clean":"2024-03-04T15:28:27.717224+0000","last_became_active":"2024-03-04T15:27:59.464804+0000","last_became_peered":"2024-03-04T15:27:59.464804+0000","last_unstale":"2024-03-04T15:28:27.717224+0000","last_undegraded":"2024-03-04T15:28:27.717224+0000","last_fullsized":"2024-03-04T15:28:27.717224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:47:06.672369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717963+0000","last_change":"2024-03-04T15:28:01.482185+0000","last_active":"2024-03-04T15:28:27.717963+0000","last_peered":"2024-03-04T15:28:27.717963+0000","last_clean":"2024-03-04T15:28:27.717963+0000","last_became_active":"2024-03-04T15:28:01.481517+0000","last_became_peered":"2024-03-04T15:28:01.481517+0000","last_unstale":"2024-03-04T15:28:27.717963+0000","last_undegraded":"2024-03-04T15:28:27.717963+0000","last_fullsized":"2024-03-04T15:28:27.717963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:21:29.742599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718053+0000","last_change":"2024-03-04T15:28:05.497934+0000","last_active":"2024-03-04T15:28:27.718053+0000","last_peered":"2024-03-04T15:28:27.718053+0000","last_clean":"2024-03-04T15:28:27.718053+0000","last_became_active":"2024-03-04T15:28:05.497649+0000","last_became_peered":"2024-03-04T15:28:05.497649+0000","last_unstale":"2024-03-04T15:28:27.718053+0000","last_undegraded":"2024-03-04T15:28:27.718053+0000","last_fullsized":"2024-03-04T15:28:27.718053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:50.364879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716492+0000","last_change":"2024-03-04T15:28:03.486210+0000","last_active":"2024-03-04T15:28:27.716492+0000","last_peered":"2024-03-04T15:28:27.716492+0000","last_clean":"2024-03-04T15:28:27.716492+0000","last_became_active":"2024-03-04T15:28:03.485813+0000","last_became_peered":"2024-03-04T15:28:03.485813+0000","last_unstale":"2024-03-04T15:28:27.716492+0000","last_undegraded":"2024-03-04T15:28:27.716492+0000","last_fullsized":"2024-03-04T15:28:27.716492+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:07.827278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721268+0000","last_change":"2024-03-04T15:27:59.467516+0000","last_active":"2024-03-04T15:28:27.721268+0000","last_peered":"2024-03-04T15:28:27.721268+0000","last_clean":"2024-03-04T15:28:27.721268+0000","last_became_active":"2024-03-04T15:27:59.466680+0000","last_became_peered":"2024-03-04T15:27:59.466680+0000","last_unstale":"2024-03-04T15:28:27.721268+0000","last_undegraded":"2024-03-04T15:28:27.721268+0000","last_fullsized":"2024-03-04T15:28:27.721268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:37:59.565627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717567+0000","last_change":"2024-03-04T15:28:01.477631+0000","last_active":"2024-03-04T15:28:27.717567+0000","last_peered":"2024-03-04T15:28:27.717567+0000","last_clean":"2024-03-04T15:28:27.717567+0000","last_became_active":"2024-03-04T15:28:01.477303+0000","last_became_peered":"2024-03-04T15:28:01.477303+0000","last_unstale":"2024-03-04T15:28:27.717567+0000","last_undegraded":"2024-03-04T15:28:27.717567+0000","last_fullsized":"2024-03-04T15:28:27.717567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:29:46.954475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716264+0000","last_change":"2024-03-04T15:28:05.501288+0000","last_active":"2024-03-04T15:28:27.716264+0000","last_peered":"2024-03-04T15:28:27.716264+0000","last_clean":"2024-03-04T15:28:27.716264+0000","last_became_active":"2024-03-04T15:28:05.500839+0000","last_became_peered":"2024-03-04T15:28:05.500839+0000","last_unstale":"2024-03-04T15:28:27.716264+0000","last_undegraded":"2024-03-04T15:28:27.716264+0000","last_fullsized":"2024-03-04T15:28:27.716264+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:37.966183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715933+0000","last_change":"2024-03-04T15:28:03.485384+0000","last_active":"2024-03-04T15:28:27.715933+0000","last_peered":"2024-03-04T15:28:27.715933+0000","last_clean":"2024-03-04T15:28:27.715933+0000","last_became_active":"2024-03-04T15:28:03.484995+0000","last_became_peered":"2024-03-04T15:28:03.484995+0000","last_unstale":"2024-03-04T15:28:27.715933+0000","last_undegraded":"2024-03-04T15:28:27.715933+0000","last_fullsized":"2024-03-04T15:28:27.715933+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:51:27.872640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718980+0000","last_change":"2024-03-04T15:27:59.465210+0000","last_active":"2024-03-04T15:28:27.718980+0000","last_peered":"2024-03-04T15:28:27.718980+0000","last_clean":"2024-03-04T15:28:27.718980+0000","last_became_active":"2024-03-04T15:27:59.464495+0000","last_became_peered":"2024-03-04T15:27:59.464495+0000","last_unstale":"2024-03-04T15:28:27.718980+0000","last_undegraded":"2024-03-04T15:28:27.718980+0000","last_fullsized":"2024-03-04T15:28:27.718980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:02.607815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721188+0000","last_change":"2024-03-04T15:28:01.478147+0000","last_active":"2024-03-04T15:28:27.721188+0000","last_peered":"2024-03-04T15:28:27.721188+0000","last_clean":"2024-03-04T15:28:27.721188+0000","last_became_active":"2024-03-04T15:28:01.477848+0000","last_became_peered":"2024-03-04T15:28:01.477848+0000","last_unstale":"2024-03-04T15:28:27.721188+0000","last_undegraded":"2024-03-04T15:28:27.721188+0000","last_fullsized":"2024-03-04T15:28:27.721188+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:05:53.676778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716620+0000","last_change":"2024-03-04T15:28:05.502385+0000","last_active":"2024-03-04T15:28:27.716620+0000","last_peered":"2024-03-04T15:28:27.716620+0000","last_clean":"2024-03-04T15:28:27.716620+0000","last_became_active":"2024-03-04T15:28:05.501700+0000","last_became_peered":"2024-03-04T15:28:05.501700+0000","last_unstale":"2024-03-04T15:28:27.716620+0000","last_undegraded":"2024-03-04T15:28:27.716620+0000","last_fullsized":"2024-03-04T15:28:27.716620+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:36:24.250295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720879+0000","last_change":"2024-03-04T15:28:03.484195+0000","last_active":"2024-03-04T15:28:27.720879+0000","last_peered":"2024-03-04T15:28:27.720879+0000","last_clean":"2024-03-04T15:28:27.720879+0000","last_became_active":"2024-03-04T15:28:03.483881+0000","last_became_peered":"2024-03-04T15:28:03.483881+0000","last_unstale":"2024-03-04T15:28:27.720879+0000","last_undegraded":"2024-03-04T15:28:27.720879+0000","last_fullsized":"2024-03-04T15:28:27.720879+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:41:07.619198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719622+0000","last_change":"2024-03-04T15:27:59.464506+0000","last_active":"2024-03-04T15:28:27.719622+0000","last_peered":"2024-03-04T15:28:27.719622+0000","last_clean":"2024-03-04T15:28:27.719622+0000","last_became_active":"2024-03-04T15:27:59.464078+0000","last_became_peered":"2024-03-04T15:27:59.464078+0000","last_unstale":"2024-03-04T15:28:27.719622+0000","last_undegraded":"2024-03-04T15:28:27.719622+0000","last_fullsized":"2024-03-04T15:28:27.719622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:46.279408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719794+0000","last_change":"2024-03-04T15:28:01.477571+0000","last_active":"2024-03-04T15:28:27.719794+0000","last_peered":"2024-03-04T15:28:27.719794+0000","last_clean":"2024-03-04T15:28:27.719794+0000","last_became_active":"2024-03-04T15:28:01.477049+0000","last_became_peered":"2024-03-04T15:28:01.477049+0000","last_unstale":"2024-03-04T15:28:27.719794+0000","last_undegraded":"2024-03-04T15:28:27.719794+0000","last_fullsized":"2024-03-04T15:28:27.719794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:39:52.803286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715969+0000","last_change":"2024-03-04T15:28:05.500221+0000","last_active":"2024-03-04T15:28:27.715969+0000","last_peered":"2024-03-04T15:28:27.715969+0000","last_clean":"2024-03-04T15:28:27.715969+0000","last_became_active":"2024-03-04T15:28:05.499831+0000","last_became_peered":"2024-03-04T15:28:05.499831+0000","last_unstale":"2024-03-04T15:28:27.715969+0000","last_undegraded":"2024-03-04T15:28:27.715969+0000","last_fullsized":"2024-03-04T15:28:27.715969+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:43:12.736111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718766+0000","last_change":"2024-03-04T15:28:03.483624+0000","last_active":"2024-03-04T15:28:27.718766+0000","last_peered":"2024-03-04T15:28:27.718766+0000","last_clean":"2024-03-04T15:28:27.718766+0000","last_became_active":"2024-03-04T15:28:03.483198+0000","last_became_peered":"2024-03-04T15:28:03.483198+0000","last_unstale":"2024-03-04T15:28:27.718766+0000","last_undegraded":"2024-03-04T15:28:27.718766+0000","last_fullsized":"2024-03-04T15:28:27.718766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:21:43.682326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716193+0000","last_change":"2024-03-04T15:27:59.467016+0000","last_active":"2024-03-04T15:28:27.716193+0000","last_peered":"2024-03-04T15:28:27.716193+0000","last_clean":"2024-03-04T15:28:27.716193+0000","last_became_active":"2024-03-04T15:27:59.466624+0000","last_became_peered":"2024-03-04T15:27:59.466624+0000","last_unstale":"2024-03-04T15:28:27.716193+0000","last_undegraded":"2024-03-04T15:28:27.716193+0000","last_fullsized":"2024-03-04T15:28:27.716193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:37:30.676021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719442+0000","last_change":"2024-03-04T15:28:01.477226+0000","last_active":"2024-03-04T15:28:27.719442+0000","last_peered":"2024-03-04T15:28:27.719442+0000","last_clean":"2024-03-04T15:28:27.719442+0000","last_became_active":"2024-03-04T15:28:01.476893+0000","last_became_peered":"2024-03-04T15:28:01.476893+0000","last_unstale":"2024-03-04T15:28:27.719442+0000","last_undegraded":"2024-03-04T15:28:27.719442+0000","last_fullsized":"2024-03-04T15:28:27.719442+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:18:36.974992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716415+0000","last_change":"2024-03-04T15:28:05.501291+0000","last_active":"2024-03-04T15:28:27.716415+0000","last_peered":"2024-03-04T15:28:27.716415+0000","last_clean":"2024-03-04T15:28:27.716415+0000","last_became_active":"2024-03-04T15:28:05.500329+0000","last_became_peered":"2024-03-04T15:28:05.500329+0000","last_unstale":"2024-03-04T15:28:27.716415+0000","last_undegraded":"2024-03-04T15:28:27.716415+0000","last_fullsized":"2024-03-04T15:28:27.716415+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:19:15.027929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513450+0000","last_change":"2024-03-04T15:28:03.483892+0000","last_active":"2024-03-04T15:28:29.513450+0000","last_peered":"2024-03-04T15:28:29.513450+0000","last_clean":"2024-03-04T15:28:29.513450+0000","last_became_active":"2024-03-04T15:28:03.482840+0000","last_became_peered":"2024-03-04T15:28:03.482840+0000","last_unstale":"2024-03-04T15:28:29.513450+0000","last_undegraded":"2024-03-04T15:28:29.513450+0000","last_fullsized":"2024-03-04T15:28:29.513450+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:48.021076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714196+0000","last_change":"2024-03-04T15:27:59.466120+0000","last_active":"2024-03-04T15:28:27.714196+0000","last_peered":"2024-03-04T15:28:27.714196+0000","last_clean":"2024-03-04T15:28:27.714196+0000","last_became_active":"2024-03-04T15:27:59.463786+0000","last_became_peered":"2024-03-04T15:27:59.463786+0000","last_unstale":"2024-03-04T15:28:27.714196+0000","last_undegraded":"2024-03-04T15:28:27.714196+0000","last_fullsized":"2024-03-04T15:28:27.714196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:53:11.301306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713482+0000","last_change":"2024-03-04T15:28:01.478331+0000","last_active":"2024-03-04T15:28:27.713482+0000","last_peered":"2024-03-04T15:28:27.713482+0000","last_clean":"2024-03-04T15:28:27.713482+0000","last_became_active":"2024-03-04T15:28:01.477947+0000","last_became_peered":"2024-03-04T15:28:01.477947+0000","last_unstale":"2024-03-04T15:28:27.713482+0000","last_undegraded":"2024-03-04T15:28:27.713482+0000","last_fullsized":"2024-03-04T15:28:27.713482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:48:40.390358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716810+0000","last_change":"2024-03-04T15:28:05.503387+0000","last_active":"2024-03-04T15:28:27.716810+0000","last_peered":"2024-03-04T15:28:27.716810+0000","last_clean":"2024-03-04T15:28:27.716810+0000","last_became_active":"2024-03-04T15:28:05.502717+0000","last_became_peered":"2024-03-04T15:28:05.502717+0000","last_unstale":"2024-03-04T15:28:27.716810+0000","last_undegraded":"2024-03-04T15:28:27.716810+0000","last_fullsized":"2024-03-04T15:28:27.716810+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:26:00.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716575+0000","last_change":"2024-03-04T15:28:03.483366+0000","last_active":"2024-03-04T15:28:27.716575+0000","last_peered":"2024-03-04T15:28:27.716575+0000","last_clean":"2024-03-04T15:28:27.716575+0000","last_became_active":"2024-03-04T15:28:03.482751+0000","last_became_peered":"2024-03-04T15:28:03.482751+0000","last_unstale":"2024-03-04T15:28:27.716575+0000","last_undegraded":"2024-03-04T15:28:27.716575+0000","last_fullsized":"2024-03-04T15:28:27.716575+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:17:53.826743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717916+0000","last_change":"2024-03-04T15:27:59.465916+0000","last_active":"2024-03-04T15:28:27.717916+0000","last_peered":"2024-03-04T15:28:27.717916+0000","last_clean":"2024-03-04T15:28:27.717916+0000","last_became_active":"2024-03-04T15:27:59.465557+0000","last_became_peered":"2024-03-04T15:27:59.465557+0000","last_unstale":"2024-03-04T15:28:27.717916+0000","last_undegraded":"2024-03-04T15:28:27.717916+0000","last_fullsized":"2024-03-04T15:28:27.717916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:51:25.628723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713764+0000","last_change":"2024-03-04T15:28:01.475943+0000","last_active":"2024-03-04T15:28:27.713764+0000","last_peered":"2024-03-04T15:28:27.713764+0000","last_clean":"2024-03-04T15:28:27.713764+0000","last_became_active":"2024-03-04T15:28:01.475596+0000","last_became_peered":"2024-03-04T15:28:01.475596+0000","last_unstale":"2024-03-04T15:28:27.713764+0000","last_undegraded":"2024-03-04T15:28:27.713764+0000","last_fullsized":"2024-03-04T15:28:27.713764+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:50:21.697880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716139+0000","last_change":"2024-03-04T15:28:05.500105+0000","last_active":"2024-03-04T15:28:27.716139+0000","last_peered":"2024-03-04T15:28:27.716139+0000","last_clean":"2024-03-04T15:28:27.716139+0000","last_became_active":"2024-03-04T15:28:05.499740+0000","last_became_peered":"2024-03-04T15:28:05.499740+0000","last_unstale":"2024-03-04T15:28:27.716139+0000","last_undegraded":"2024-03-04T15:28:27.716139+0000","last_fullsized":"2024-03-04T15:28:27.716139+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:44:25.099992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713598+0000","last_change":"2024-03-04T15:28:03.483182+0000","last_active":"2024-03-04T15:28:27.713598+0000","last_peered":"2024-03-04T15:28:27.713598+0000","last_clean":"2024-03-04T15:28:27.713598+0000","last_became_active":"2024-03-04T15:28:03.482745+0000","last_became_peered":"2024-03-04T15:28:03.482745+0000","last_unstale":"2024-03-04T15:28:27.713598+0000","last_undegraded":"2024-03-04T15:28:27.713598+0000","last_fullsized":"2024-03-04T15:28:27.713598+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:35.644553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718552+0000","last_change":"2024-03-04T15:27:59.468542+0000","last_active":"2024-03-04T15:28:27.718552+0000","last_peered":"2024-03-04T15:28:27.718552+0000","last_clean":"2024-03-04T15:28:27.718552+0000","last_became_active":"2024-03-04T15:27:59.468144+0000","last_became_peered":"2024-03-04T15:27:59.468144+0000","last_unstale":"2024-03-04T15:28:27.718552+0000","last_undegraded":"2024-03-04T15:28:27.718552+0000","last_fullsized":"2024-03-04T15:28:27.718552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:31.313829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717582+0000","last_change":"2024-03-04T15:28:01.479383+0000","last_active":"2024-03-04T15:28:27.717582+0000","last_peered":"2024-03-04T15:28:27.717582+0000","last_clean":"2024-03-04T15:28:27.717582+0000","last_became_active":"2024-03-04T15:28:01.478835+0000","last_became_peered":"2024-03-04T15:28:01.478835+0000","last_unstale":"2024-03-04T15:28:27.717582+0000","last_undegraded":"2024-03-04T15:28:27.717582+0000","last_fullsized":"2024-03-04T15:28:27.717582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:54:31.667939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716122+0000","last_change":"2024-03-04T15:28:05.502051+0000","last_active":"2024-03-04T15:28:27.716122+0000","last_peered":"2024-03-04T15:28:27.716122+0000","last_clean":"2024-03-04T15:28:27.716122+0000","last_became_active":"2024-03-04T15:28:05.500240+0000","last_became_peered":"2024-03-04T15:28:05.500240+0000","last_unstale":"2024-03-04T15:28:27.716122+0000","last_undegraded":"2024-03-04T15:28:27.716122+0000","last_fullsized":"2024-03-04T15:28:27.716122+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:55:26.833913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713966+0000","last_change":"2024-03-04T15:28:03.483169+0000","last_active":"2024-03-04T15:28:27.713966+0000","last_peered":"2024-03-04T15:28:27.713966+0000","last_clean":"2024-03-04T15:28:27.713966+0000","last_became_active":"2024-03-04T15:28:03.482660+0000","last_became_peered":"2024-03-04T15:28:03.482660+0000","last_unstale":"2024-03-04T15:28:27.713966+0000","last_undegraded":"2024-03-04T15:28:27.713966+0000","last_fullsized":"2024-03-04T15:28:27.713966+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:15:47.790176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717326+0000","last_change":"2024-03-04T15:27:59.469994+0000","last_active":"2024-03-04T15:28:27.717326+0000","last_peered":"2024-03-04T15:28:27.717326+0000","last_clean":"2024-03-04T15:28:27.717326+0000","last_became_active":"2024-03-04T15:27:59.469245+0000","last_became_peered":"2024-03-04T15:27:59.469245+0000","last_unstale":"2024-03-04T15:28:27.717326+0000","last_undegraded":"2024-03-04T15:28:27.717326+0000","last_fullsized":"2024-03-04T15:28:27.717326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:53:34.199893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718044+0000","last_change":"2024-03-04T15:28:01.481892+0000","last_active":"2024-03-04T15:28:27.718044+0000","last_peered":"2024-03-04T15:28:27.718044+0000","last_clean":"2024-03-04T15:28:27.718044+0000","last_became_active":"2024-03-04T15:28:01.481508+0000","last_became_peered":"2024-03-04T15:28:01.481508+0000","last_unstale":"2024-03-04T15:28:27.718044+0000","last_undegraded":"2024-03-04T15:28:27.718044+0000","last_fullsized":"2024-03-04T15:28:27.718044+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:49:17.811926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719023+0000","last_change":"2024-03-04T15:28:05.498049+0000","last_active":"2024-03-04T15:28:27.719023+0000","last_peered":"2024-03-04T15:28:27.719023+0000","last_clean":"2024-03-04T15:28:27.719023+0000","last_became_active":"2024-03-04T15:28:05.497801+0000","last_became_peered":"2024-03-04T15:28:05.497801+0000","last_unstale":"2024-03-04T15:28:27.719023+0000","last_undegraded":"2024-03-04T15:28:27.719023+0000","last_fullsized":"2024-03-04T15:28:27.719023+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:03:55.560117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716918+0000","last_change":"2024-03-04T15:28:03.484029+0000","last_active":"2024-03-04T15:28:27.716918+0000","last_peered":"2024-03-04T15:28:27.716918+0000","last_clean":"2024-03-04T15:28:27.716918+0000","last_became_active":"2024-03-04T15:28:03.482746+0000","last_became_peered":"2024-03-04T15:28:03.482746+0000","last_unstale":"2024-03-04T15:28:27.716918+0000","last_undegraded":"2024-03-04T15:28:27.716918+0000","last_fullsized":"2024-03-04T15:28:27.716918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:58.224350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721183+0000","last_change":"2024-03-04T15:27:59.467797+0000","last_active":"2024-03-04T15:28:27.721183+0000","last_peered":"2024-03-04T15:28:27.721183+0000","last_clean":"2024-03-04T15:28:27.721183+0000","last_became_active":"2024-03-04T15:27:59.466906+0000","last_became_peered":"2024-03-04T15:27:59.466906+0000","last_unstale":"2024-03-04T15:28:27.721183+0000","last_undegraded":"2024-03-04T15:28:27.721183+0000","last_fullsized":"2024-03-04T15:28:27.721183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:14:38.401121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717418+0000","last_change":"2024-03-04T15:28:01.479770+0000","last_active":"2024-03-04T15:28:27.717418+0000","last_peered":"2024-03-04T15:28:27.717418+0000","last_clean":"2024-03-04T15:28:27.717418+0000","last_became_active":"2024-03-04T15:28:01.479195+0000","last_became_peered":"2024-03-04T15:28:01.479195+0000","last_unstale":"2024-03-04T15:28:27.717418+0000","last_undegraded":"2024-03-04T15:28:27.717418+0000","last_fullsized":"2024-03-04T15:28:27.717418+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:02:49.179079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720368+0000","last_change":"2024-03-04T15:28:05.497142+0000","last_active":"2024-03-04T15:28:27.720368+0000","last_peered":"2024-03-04T15:28:27.720368+0000","last_clean":"2024-03-04T15:28:27.720368+0000","last_became_active":"2024-03-04T15:28:05.496176+0000","last_became_peered":"2024-03-04T15:28:05.496176+0000","last_unstale":"2024-03-04T15:28:27.720368+0000","last_undegraded":"2024-03-04T15:28:27.720368+0000","last_fullsized":"2024-03-04T15:28:27.720368+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:44:55.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713752+0000","last_change":"2024-03-04T15:28:03.482722+0000","last_active":"2024-03-04T15:28:27.713752+0000","last_peered":"2024-03-04T15:28:27.713752+0000","last_clean":"2024-03-04T15:28:27.713752+0000","last_became_active":"2024-03-04T15:28:03.482340+0000","last_became_peered":"2024-03-04T15:28:03.482340+0000","last_unstale":"2024-03-04T15:28:27.713752+0000","last_undegraded":"2024-03-04T15:28:27.713752+0000","last_fullsized":"2024-03-04T15:28:27.713752+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:25:43.423784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717074+0000","last_change":"2024-03-04T15:27:59.467849+0000","last_active":"2024-03-04T15:28:27.717074+0000","last_peered":"2024-03-04T15:28:27.717074+0000","last_clean":"2024-03-04T15:28:27.717074+0000","last_became_active":"2024-03-04T15:27:59.466835+0000","last_became_peered":"2024-03-04T15:27:59.466835+0000","last_unstale":"2024-03-04T15:28:27.717074+0000","last_undegraded":"2024-03-04T15:28:27.717074+0000","last_fullsized":"2024-03-04T15:28:27.717074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:43.023274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721034+0000","last_change":"2024-03-04T15:28:01.477243+0000","last_active":"2024-03-04T15:28:27.721034+0000","last_peered":"2024-03-04T15:28:27.721034+0000","last_clean":"2024-03-04T15:28:27.721034+0000","last_became_active":"2024-03-04T15:28:01.476903+0000","last_became_peered":"2024-03-04T15:28:01.476903+0000","last_unstale":"2024-03-04T15:28:27.721034+0000","last_undegraded":"2024-03-04T15:28:27.721034+0000","last_fullsized":"2024-03-04T15:28:27.721034+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:12:26.329137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716983+0000","last_change":"2024-03-04T15:28:05.500635+0000","last_active":"2024-03-04T15:28:27.716983+0000","last_peered":"2024-03-04T15:28:27.716983+0000","last_clean":"2024-03-04T15:28:27.716983+0000","last_became_active":"2024-03-04T15:28:05.500042+0000","last_became_peered":"2024-03-04T15:28:05.500042+0000","last_unstale":"2024-03-04T15:28:27.716983+0000","last_undegraded":"2024-03-04T15:28:27.716983+0000","last_fullsized":"2024-03-04T15:28:27.716983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:30.938794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513466+0000","last_change":"2024-03-04T15:28:03.483434+0000","last_active":"2024-03-04T15:28:29.513466+0000","last_peered":"2024-03-04T15:28:29.513466+0000","last_clean":"2024-03-04T15:28:29.513466+0000","last_became_active":"2024-03-04T15:28:03.482734+0000","last_became_peered":"2024-03-04T15:28:03.482734+0000","last_unstale":"2024-03-04T15:28:29.513466+0000","last_undegraded":"2024-03-04T15:28:29.513466+0000","last_fullsized":"2024-03-04T15:28:29.513466+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:29:32.455559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718992+0000","last_change":"2024-03-04T15:27:59.464412+0000","last_active":"2024-03-04T15:28:27.718992+0000","last_peered":"2024-03-04T15:28:27.718992+0000","last_clean":"2024-03-04T15:28:27.718992+0000","last_became_active":"2024-03-04T15:27:59.464049+0000","last_became_peered":"2024-03-04T15:27:59.464049+0000","last_unstale":"2024-03-04T15:28:27.718992+0000","last_undegraded":"2024-03-04T15:28:27.718992+0000","last_fullsized":"2024-03-04T15:28:27.718992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:48:26.542755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715467+0000","last_change":"2024-03-04T15:28:01.477596+0000","last_active":"2024-03-04T15:28:27.715467+0000","last_peered":"2024-03-04T15:28:27.715467+0000","last_clean":"2024-03-04T15:28:27.715467+0000","last_became_active":"2024-03-04T15:28:01.477080+0000","last_became_peered":"2024-03-04T15:28:01.477080+0000","last_unstale":"2024-03-04T15:28:27.715467+0000","last_undegraded":"2024-03-04T15:28:27.715467+0000","last_fullsized":"2024-03-04T15:28:27.715467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:38:26.075233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717301+0000","last_change":"2024-03-04T15:28:05.502039+0000","last_active":"2024-03-04T15:28:27.717301+0000","last_peered":"2024-03-04T15:28:27.717301+0000","last_clean":"2024-03-04T15:28:27.717301+0000","last_became_active":"2024-03-04T15:28:05.501453+0000","last_became_peered":"2024-03-04T15:28:05.501453+0000","last_unstale":"2024-03-04T15:28:27.717301+0000","last_undegraded":"2024-03-04T15:28:27.717301+0000","last_fullsized":"2024-03-04T15:28:27.717301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:00:43.927613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513484+0000","last_change":"2024-03-04T15:28:03.483909+0000","last_active":"2024-03-04T15:28:29.513484+0000","last_peered":"2024-03-04T15:28:29.513484+0000","last_clean":"2024-03-04T15:28:29.513484+0000","last_became_active":"2024-03-04T15:28:03.483040+0000","last_became_peered":"2024-03-04T15:28:03.483040+0000","last_unstale":"2024-03-04T15:28:29.513484+0000","last_undegraded":"2024-03-04T15:28:29.513484+0000","last_fullsized":"2024-03-04T15:28:29.513484+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:18:24.067938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719383+0000","last_change":"2024-03-04T15:27:59.465444+0000","last_active":"2024-03-04T15:28:27.719383+0000","last_peered":"2024-03-04T15:28:27.719383+0000","last_clean":"2024-03-04T15:28:27.719383+0000","last_became_active":"2024-03-04T15:27:59.464144+0000","last_became_peered":"2024-03-04T15:27:59.464144+0000","last_unstale":"2024-03-04T15:28:27.719383+0000","last_undegraded":"2024-03-04T15:28:27.719383+0000","last_fullsized":"2024-03-04T15:28:27.719383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:02:11.715204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717406+0000","last_change":"2024-03-04T15:28:01.477055+0000","last_active":"2024-03-04T15:28:27.717406+0000","last_peered":"2024-03-04T15:28:27.717406+0000","last_clean":"2024-03-04T15:28:27.717406+0000","last_became_active":"2024-03-04T15:28:01.475829+0000","last_became_peered":"2024-03-04T15:28:01.475829+0000","last_unstale":"2024-03-04T15:28:27.717406+0000","last_undegraded":"2024-03-04T15:28:27.717406+0000","last_fullsized":"2024-03-04T15:28:27.717406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:49:43.088367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716859+0000","last_change":"2024-03-04T15:28:05.502063+0000","last_active":"2024-03-04T15:28:27.716859+0000","last_peered":"2024-03-04T15:28:27.716859+0000","last_clean":"2024-03-04T15:28:27.716859+0000","last_became_active":"2024-03-04T15:28:05.501410+0000","last_became_peered":"2024-03-04T15:28:05.501410+0000","last_unstale":"2024-03-04T15:28:27.716859+0000","last_undegraded":"2024-03-04T15:28:27.716859+0000","last_fullsized":"2024-03-04T15:28:27.716859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:32.805593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717896+0000","last_change":"2024-03-04T15:28:03.487646+0000","last_active":"2024-03-04T15:28:27.717896+0000","last_peered":"2024-03-04T15:28:27.717896+0000","last_clean":"2024-03-04T15:28:27.717896+0000","last_became_active":"2024-03-04T15:28:03.487244+0000","last_became_peered":"2024-03-04T15:28:03.487244+0000","last_unstale":"2024-03-04T15:28:27.717896+0000","last_undegraded":"2024-03-04T15:28:27.717896+0000","last_fullsized":"2024-03-04T15:28:27.717896+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:33.760668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715865+0000","last_change":"2024-03-04T15:27:59.467335+0000","last_active":"2024-03-04T15:28:27.715865+0000","last_peered":"2024-03-04T15:28:27.715865+0000","last_clean":"2024-03-04T15:28:27.715865+0000","last_became_active":"2024-03-04T15:27:59.466401+0000","last_became_peered":"2024-03-04T15:27:59.466401+0000","last_unstale":"2024-03-04T15:28:27.715865+0000","last_undegraded":"2024-03-04T15:28:27.715865+0000","last_fullsized":"2024-03-04T15:28:27.715865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:13.079130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715494+0000","last_change":"2024-03-04T15:28:01.476066+0000","last_active":"2024-03-04T15:28:27.715494+0000","last_peered":"2024-03-04T15:28:27.715494+0000","last_clean":"2024-03-04T15:28:27.715494+0000","last_became_active":"2024-03-04T15:28:01.475619+0000","last_became_peered":"2024-03-04T15:28:01.475619+0000","last_unstale":"2024-03-04T15:28:27.715494+0000","last_undegraded":"2024-03-04T15:28:27.715494+0000","last_fullsized":"2024-03-04T15:28:27.715494+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:56:51.312084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717673+0000","last_change":"2024-03-04T15:28:05.501493+0000","last_active":"2024-03-04T15:28:27.717673+0000","last_peered":"2024-03-04T15:28:27.717673+0000","last_clean":"2024-03-04T15:28:27.717673+0000","last_became_active":"2024-03-04T15:28:05.501135+0000","last_became_peered":"2024-03-04T15:28:05.501135+0000","last_unstale":"2024-03-04T15:28:27.717673+0000","last_undegraded":"2024-03-04T15:28:27.717673+0000","last_fullsized":"2024-03-04T15:28:27.717673+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:31:33.148825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716663+0000","last_change":"2024-03-04T15:28:03.487580+0000","last_active":"2024-03-04T15:28:27.716663+0000","last_peered":"2024-03-04T15:28:27.716663+0000","last_clean":"2024-03-04T15:28:27.716663+0000","last_became_active":"2024-03-04T15:28:03.487083+0000","last_became_peered":"2024-03-04T15:28:03.487083+0000","last_unstale":"2024-03-04T15:28:27.716663+0000","last_undegraded":"2024-03-04T15:28:27.716663+0000","last_fullsized":"2024-03-04T15:28:27.716663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:07:22.311701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719705+0000","last_change":"2024-03-04T15:27:59.465840+0000","last_active":"2024-03-04T15:28:27.719705+0000","last_peered":"2024-03-04T15:28:27.719705+0000","last_clean":"2024-03-04T15:28:27.719705+0000","last_became_active":"2024-03-04T15:27:59.465362+0000","last_became_peered":"2024-03-04T15:27:59.465362+0000","last_unstale":"2024-03-04T15:28:27.719705+0000","last_undegraded":"2024-03-04T15:28:27.719705+0000","last_fullsized":"2024-03-04T15:28:27.719705+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.791581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718251+0000","last_change":"2024-03-04T15:28:01.477714+0000","last_active":"2024-03-04T15:28:27.718251+0000","last_peered":"2024-03-04T15:28:27.718251+0000","last_clean":"2024-03-04T15:28:27.718251+0000","last_became_active":"2024-03-04T15:28:01.477273+0000","last_became_peered":"2024-03-04T15:28:01.477273+0000","last_unstale":"2024-03-04T15:28:27.718251+0000","last_undegraded":"2024-03-04T15:28:27.718251+0000","last_fullsized":"2024-03-04T15:28:27.718251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:24:40.858313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717636+0000","last_change":"2024-03-04T15:28:05.497141+0000","last_active":"2024-03-04T15:28:27.717636+0000","last_peered":"2024-03-04T15:28:27.717636+0000","last_clean":"2024-03-04T15:28:27.717636+0000","last_became_active":"2024-03-04T15:28:05.496285+0000","last_became_peered":"2024-03-04T15:28:05.496285+0000","last_unstale":"2024-03-04T15:28:27.717636+0000","last_undegraded":"2024-03-04T15:28:27.717636+0000","last_fullsized":"2024-03-04T15:28:27.717636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:06.238963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720621+0000","last_change":"2024-03-04T15:27:59.467227+0000","last_active":"2024-03-04T15:28:27.720621+0000","last_peered":"2024-03-04T15:28:27.720621+0000","last_clean":"2024-03-04T15:28:27.720621+0000","last_became_active":"2024-03-04T15:27:59.466669+0000","last_became_peered":"2024-03-04T15:27:59.466669+0000","last_unstale":"2024-03-04T15:28:27.720621+0000","last_undegraded":"2024-03-04T15:28:27.720621+0000","last_fullsized":"2024-03-04T15:28:27.720621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:23:57.085399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719407+0000","last_change":"2024-03-04T15:28:01.478022+0000","last_active":"2024-03-04T15:28:27.719407+0000","last_peered":"2024-03-04T15:28:27.719407+0000","last_clean":"2024-03-04T15:28:27.719407+0000","last_became_active":"2024-03-04T15:28:01.477701+0000","last_became_peered":"2024-03-04T15:28:01.477701+0000","last_unstale":"2024-03-04T15:28:27.719407+0000","last_undegraded":"2024-03-04T15:28:27.719407+0000","last_fullsized":"2024-03-04T15:28:27.719407+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:19.988069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513507+0000","last_change":"2024-03-04T15:28:03.483828+0000","last_active":"2024-03-04T15:28:29.513507+0000","last_peered":"2024-03-04T15:28:29.513507+0000","last_clean":"2024-03-04T15:28:29.513507+0000","last_became_active":"2024-03-04T15:28:03.483535+0000","last_became_peered":"2024-03-04T15:28:03.483535+0000","last_unstale":"2024-03-04T15:28:29.513507+0000","last_undegraded":"2024-03-04T15:28:29.513507+0000","last_fullsized":"2024-03-04T15:28:29.513507+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:34.065643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714234+0000","last_change":"2024-03-04T15:28:05.497753+0000","last_active":"2024-03-04T15:28:27.714234+0000","last_peered":"2024-03-04T15:28:27.714234+0000","last_clean":"2024-03-04T15:28:27.714234+0000","last_became_active":"2024-03-04T15:28:05.497343+0000","last_became_peered":"2024-03-04T15:28:05.497343+0000","last_unstale":"2024-03-04T15:28:27.714234+0000","last_undegraded":"2024-03-04T15:28:27.714234+0000","last_fullsized":"2024-03-04T15:28:27.714234+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:30.610429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717291+0000","last_change":"2024-03-04T15:27:59.464692+0000","last_active":"2024-03-04T15:28:27.717291+0000","last_peered":"2024-03-04T15:28:27.717291+0000","last_clean":"2024-03-04T15:28:27.717291+0000","last_became_active":"2024-03-04T15:27:59.463639+0000","last_became_peered":"2024-03-04T15:27:59.463639+0000","last_unstale":"2024-03-04T15:28:27.717291+0000","last_undegraded":"2024-03-04T15:28:27.717291+0000","last_fullsized":"2024-03-04T15:28:27.717291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:54:55.898856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513395+0000","last_change":"2024-03-04T15:28:01.476694+0000","last_active":"2024-03-04T15:28:29.513395+0000","last_peered":"2024-03-04T15:28:29.513395+0000","last_clean":"2024-03-04T15:28:29.513395+0000","last_became_active":"2024-03-04T15:28:01.475719+0000","last_became_peered":"2024-03-04T15:28:01.475719+0000","last_unstale":"2024-03-04T15:28:29.513395+0000","last_undegraded":"2024-03-04T15:28:29.513395+0000","last_fullsized":"2024-03-04T15:28:29.513395+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:42:00.331170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719144+0000","last_change":"2024-03-04T15:28:03.483112+0000","last_active":"2024-03-04T15:28:27.719144+0000","last_peered":"2024-03-04T15:28:27.719144+0000","last_clean":"2024-03-04T15:28:27.719144+0000","last_became_active":"2024-03-04T15:28:03.482821+0000","last_became_peered":"2024-03-04T15:28:03.482821+0000","last_unstale":"2024-03-04T15:28:27.719144+0000","last_undegraded":"2024-03-04T15:28:27.719144+0000","last_fullsized":"2024-03-04T15:28:27.719144+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:57:19.759092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715448+0000","last_change":"2024-03-04T15:28:05.498126+0000","last_active":"2024-03-04T15:28:27.715448+0000","last_peered":"2024-03-04T15:28:27.715448+0000","last_clean":"2024-03-04T15:28:27.715448+0000","last_became_active":"2024-03-04T15:28:05.497577+0000","last_became_peered":"2024-03-04T15:28:05.497577+0000","last_unstale":"2024-03-04T15:28:27.715448+0000","last_undegraded":"2024-03-04T15:28:27.715448+0000","last_fullsized":"2024-03-04T15:28:27.715448+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:23:24.187045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716375+0000","last_change":"2024-03-04T15:27:59.469243+0000","last_active":"2024-03-04T15:28:27.716375+0000","last_peered":"2024-03-04T15:28:27.716375+0000","last_clean":"2024-03-04T15:28:27.716375+0000","last_became_active":"2024-03-04T15:27:59.468655+0000","last_became_peered":"2024-03-04T15:27:59.468655+0000","last_unstale":"2024-03-04T15:28:27.716375+0000","last_undegraded":"2024-03-04T15:28:27.716375+0000","last_fullsized":"2024-03-04T15:28:27.716375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:50:13.408312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717533+0000","last_change":"2024-03-04T15:28:01.477060+0000","last_active":"2024-03-04T15:28:27.717533+0000","last_peered":"2024-03-04T15:28:27.717533+0000","last_clean":"2024-03-04T15:28:27.717533+0000","last_became_active":"2024-03-04T15:28:01.476214+0000","last_became_peered":"2024-03-04T15:28:01.476214+0000","last_unstale":"2024-03-04T15:28:27.717533+0000","last_undegraded":"2024-03-04T15:28:27.717533+0000","last_fullsized":"2024-03-04T15:28:27.717533+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:52:45.695981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717075+0000","last_change":"2024-03-04T15:28:03.483233+0000","last_active":"2024-03-04T15:28:27.717075+0000","last_peered":"2024-03-04T15:28:27.717075+0000","last_clean":"2024-03-04T15:28:27.717075+0000","last_became_active":"2024-03-04T15:28:03.482737+0000","last_became_peered":"2024-03-04T15:28:03.482737+0000","last_unstale":"2024-03-04T15:28:27.717075+0000","last_undegraded":"2024-03-04T15:28:27.717075+0000","last_fullsized":"2024-03-04T15:28:27.717075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:53.633584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717473+0000","last_change":"2024-03-04T15:28:05.501456+0000","last_active":"2024-03-04T15:28:27.717473+0000","last_peered":"2024-03-04T15:28:27.717473+0000","last_clean":"2024-03-04T15:28:27.717473+0000","last_became_active":"2024-03-04T15:28:05.501099+0000","last_became_peered":"2024-03-04T15:28:05.501099+0000","last_unstale":"2024-03-04T15:28:27.717473+0000","last_undegraded":"2024-03-04T15:28:27.717473+0000","last_fullsized":"2024-03-04T15:28:27.717473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:43.295522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717727+0000","last_change":"2024-03-04T15:28:01.478401+0000","last_active":"2024-03-04T15:28:27.717727+0000","last_peered":"2024-03-04T15:28:27.717727+0000","last_clean":"2024-03-04T15:28:27.717727+0000","last_became_active":"2024-03-04T15:28:01.478084+0000","last_became_peered":"2024-03-04T15:28:01.478084+0000","last_unstale":"2024-03-04T15:28:27.717727+0000","last_undegraded":"2024-03-04T15:28:27.717727+0000","last_fullsized":"2024-03-04T15:28:27.717727+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:42.662935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713947+0000","last_change":"2024-03-04T15:27:59.466334+0000","last_active":"2024-03-04T15:28:27.713947+0000","last_peered":"2024-03-04T15:28:27.713947+0000","last_clean":"2024-03-04T15:28:27.713947+0000","last_became_active":"2024-03-04T15:27:59.465459+0000","last_became_peered":"2024-03-04T15:27:59.465459+0000","last_unstale":"2024-03-04T15:28:27.713947+0000","last_undegraded":"2024-03-04T15:28:27.713947+0000","last_fullsized":"2024-03-04T15:28:27.713947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:01:52.266865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718033+0000","last_change":"2024-03-04T15:28:03.488694+0000","last_active":"2024-03-04T15:28:27.718033+0000","last_peered":"2024-03-04T15:28:27.718033+0000","last_clean":"2024-03-04T15:28:27.718033+0000","last_became_active":"2024-03-04T15:28:03.488408+0000","last_became_peered":"2024-03-04T15:28:03.488408+0000","last_unstale":"2024-03-04T15:28:27.718033+0000","last_undegraded":"2024-03-04T15:28:27.718033+0000","last_fullsized":"2024-03-04T15:28:27.718033+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:17:47.930500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718217+0000","last_change":"2024-03-04T15:28:05.500191+0000","last_active":"2024-03-04T15:28:27.718217+0000","last_peered":"2024-03-04T15:28:27.718217+0000","last_clean":"2024-03-04T15:28:27.718217+0000","last_became_active":"2024-03-04T15:28:05.499715+0000","last_became_peered":"2024-03-04T15:28:05.499715+0000","last_unstale":"2024-03-04T15:28:27.718217+0000","last_undegraded":"2024-03-04T15:28:27.718217+0000","last_fullsized":"2024-03-04T15:28:27.718217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:14.932681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715166+0000","last_change":"2024-03-04T15:28:01.475816+0000","last_active":"2024-03-04T15:28:27.715166+0000","last_peered":"2024-03-04T15:28:27.715166+0000","last_clean":"2024-03-04T15:28:27.715166+0000","last_became_active":"2024-03-04T15:28:01.475240+0000","last_became_peered":"2024-03-04T15:28:01.475240+0000","last_unstale":"2024-03-04T15:28:27.715166+0000","last_undegraded":"2024-03-04T15:28:27.715166+0000","last_fullsized":"2024-03-04T15:28:27.715166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:25.250136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717163+0000","last_change":"2024-03-04T15:27:59.467719+0000","last_active":"2024-03-04T15:28:27.717163+0000","last_peered":"2024-03-04T15:28:27.717163+0000","last_clean":"2024-03-04T15:28:27.717163+0000","last_became_active":"2024-03-04T15:27:59.467050+0000","last_became_peered":"2024-03-04T15:27:59.467050+0000","last_unstale":"2024-03-04T15:28:27.717163+0000","last_undegraded":"2024-03-04T15:28:27.717163+0000","last_fullsized":"2024-03-04T15:28:27.717163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:51:20.688294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.832660+0000","last_change":"2024-03-04T15:27:47.908050+0000","last_active":"2024-03-04T15:28:29.832660+0000","last_peered":"2024-03-04T15:28:29.832660+0000","last_clean":"2024-03-04T15:28:29.832660+0000","last_became_active":"2024-03-04T15:27:47.904891+0000","last_became_peered":"2024-03-04T15:27:47.904891+0000","last_unstale":"2024-03-04T15:28:29.832660+0000","last_undegraded":"2024-03-04T15:28:29.832660+0000","last_fullsized":"2024-03-04T15:28:29.832660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:46.888899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:46.888899+0000","last_clean_scrub_stamp":"2024-03-04T15:27:46.888899+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:26:17.050613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718379+0000","last_change":"2024-03-04T15:28:03.483931+0000","last_active":"2024-03-04T15:28:27.718379+0000","last_peered":"2024-03-04T15:28:27.718379+0000","last_clean":"2024-03-04T15:28:27.718379+0000","last_became_active":"2024-03-04T15:28:03.482483+0000","last_became_peered":"2024-03-04T15:28:03.482483+0000","last_unstale":"2024-03-04T15:28:27.718379+0000","last_undegraded":"2024-03-04T15:28:27.718379+0000","last_fullsized":"2024-03-04T15:28:27.718379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:51:35.527999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717090+0000","last_change":"2024-03-04T15:28:05.503166+0000","last_active":"2024-03-04T15:28:27.717090+0000","last_peered":"2024-03-04T15:28:27.717090+0000","last_clean":"2024-03-04T15:28:27.717090+0000","last_became_active":"2024-03-04T15:28:05.502068+0000","last_became_peered":"2024-03-04T15:28:05.502068+0000","last_unstale":"2024-03-04T15:28:27.717090+0000","last_undegraded":"2024-03-04T15:28:27.717090+0000","last_fullsized":"2024-03-04T15:28:27.717090+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:51:38.165836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714353+0000","last_change":"2024-03-04T15:28:01.478497+0000","last_active":"2024-03-04T15:28:27.714353+0000","last_peered":"2024-03-04T15:28:27.714353+0000","last_clean":"2024-03-04T15:28:27.714353+0000","last_became_active":"2024-03-04T15:28:01.478052+0000","last_became_peered":"2024-03-04T15:28:01.478052+0000","last_unstale":"2024-03-04T15:28:27.714353+0000","last_undegraded":"2024-03-04T15:28:27.714353+0000","last_fullsized":"2024-03-04T15:28:27.714353+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:41:18.463499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715501+0000","last_change":"2024-03-04T15:27:59.468589+0000","last_active":"2024-03-04T15:28:27.715501+0000","last_peered":"2024-03-04T15:28:27.715501+0000","last_clean":"2024-03-04T15:28:27.715501+0000","last_became_active":"2024-03-04T15:27:59.468254+0000","last_became_peered":"2024-03-04T15:27:59.468254+0000","last_unstale":"2024-03-04T15:28:27.715501+0000","last_undegraded":"2024-03-04T15:28:27.715501+0000","last_fullsized":"2024-03-04T15:28:27.715501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:28:56.422093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718731+0000","last_change":"2024-03-04T15:28:03.484421+0000","last_active":"2024-03-04T15:28:27.718731+0000","last_peered":"2024-03-04T15:28:27.718731+0000","last_clean":"2024-03-04T15:28:27.718731+0000","last_became_active":"2024-03-04T15:28:03.483959+0000","last_became_peered":"2024-03-04T15:28:03.483959+0000","last_unstale":"2024-03-04T15:28:27.718731+0000","last_undegraded":"2024-03-04T15:28:27.718731+0000","last_fullsized":"2024-03-04T15:28:27.718731+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:17:47.138144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716180+0000","last_change":"2024-03-04T15:28:05.501152+0000","last_active":"2024-03-04T15:28:27.716180+0000","last_peered":"2024-03-04T15:28:27.716180+0000","last_clean":"2024-03-04T15:28:27.716180+0000","last_became_active":"2024-03-04T15:28:05.500508+0000","last_became_peered":"2024-03-04T15:28:05.500508+0000","last_unstale":"2024-03-04T15:28:27.716180+0000","last_undegraded":"2024-03-04T15:28:27.716180+0000","last_fullsized":"2024-03-04T15:28:27.716180+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:03:48.967046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721120+0000","last_change":"2024-03-04T15:27:59.467864+0000","last_active":"2024-03-04T15:28:27.721120+0000","last_peered":"2024-03-04T15:28:27.721120+0000","last_clean":"2024-03-04T15:28:27.721120+0000","last_became_active":"2024-03-04T15:27:59.467177+0000","last_became_peered":"2024-03-04T15:27:59.467177+0000","last_unstale":"2024-03-04T15:28:27.721120+0000","last_undegraded":"2024-03-04T15:28:27.721120+0000","last_fullsized":"2024-03-04T15:28:27.721120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:07.360073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513311+0000","last_change":"2024-03-04T15:28:01.477896+0000","last_active":"2024-03-04T15:28:29.513311+0000","last_peered":"2024-03-04T15:28:29.513311+0000","last_clean":"2024-03-04T15:28:29.513311+0000","last_became_active":"2024-03-04T15:28:01.477412+0000","last_became_peered":"2024-03-04T15:28:01.477412+0000","last_unstale":"2024-03-04T15:28:29.513311+0000","last_undegraded":"2024-03-04T15:28:29.513311+0000","last_fullsized":"2024-03-04T15:28:29.513311+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:15.754629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513477+0000","last_change":"2024-03-04T15:28:03.484105+0000","last_active":"2024-03-04T15:28:29.513477+0000","last_peered":"2024-03-04T15:28:29.513477+0000","last_clean":"2024-03-04T15:28:29.513477+0000","last_became_active":"2024-03-04T15:28:03.483337+0000","last_became_peered":"2024-03-04T15:28:03.483337+0000","last_unstale":"2024-03-04T15:28:29.513477+0000","last_undegraded":"2024-03-04T15:28:29.513477+0000","last_fullsized":"2024-03-04T15:28:29.513477+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:02:09.706001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715644+0000","last_change":"2024-03-04T15:28:05.501304+0000","last_active":"2024-03-04T15:28:27.715644+0000","last_peered":"2024-03-04T15:28:27.715644+0000","last_clean":"2024-03-04T15:28:27.715644+0000","last_became_active":"2024-03-04T15:28:05.500387+0000","last_became_peered":"2024-03-04T15:28:05.500387+0000","last_unstale":"2024-03-04T15:28:27.715644+0000","last_undegraded":"2024-03-04T15:28:27.715644+0000","last_fullsized":"2024-03-04T15:28:27.715644+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:56:15.006474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'2","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717371+0000","last_change":"2024-03-04T15:27:59.465923+0000","last_active":"2024-03-04T15:28:27.717371+0000","last_peered":"2024-03-04T15:28:27.717371+0000","last_clean":"2024-03-04T15:28:27.717371+0000","last_became_active":"2024-03-04T15:27:59.464933+0000","last_became_peered":"2024-03-04T15:27:59.464933+0000","last_unstale":"2024-03-04T15:28:27.717371+0000","last_undegraded":"2024-03-04T15:28:27.717371+0000","last_fullsized":"2024-03-04T15:28:27.717371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:34:45.241543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":921,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718232+0000","last_change":"2024-03-04T15:28:01.482357+0000","last_active":"2024-03-04T15:28:27.718232+0000","last_peered":"2024-03-04T15:28:27.718232+0000","last_clean":"2024-03-04T15:28:27.718232+0000","last_became_active":"2024-03-04T15:28:01.482041+0000","last_became_peered":"2024-03-04T15:28:01.482041+0000","last_unstale":"2024-03-04T15:28:27.718232+0000","last_undegraded":"2024-03-04T15:28:27.718232+0000","last_fullsized":"2024-03-04T15:28:27.718232+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:30.669292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513239+0000","last_change":"2024-03-04T15:28:03.487302+0000","last_active":"2024-03-04T15:28:29.513239+0000","last_peered":"2024-03-04T15:28:29.513239+0000","last_clean":"2024-03-04T15:28:29.513239+0000","last_became_active":"2024-03-04T15:28:03.486924+0000","last_became_peered":"2024-03-04T15:28:03.486924+0000","last_unstale":"2024-03-04T15:28:29.513239+0000","last_undegraded":"2024-03-04T15:28:29.513239+0000","last_fullsized":"2024-03-04T15:28:29.513239+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:08:27.367873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719068+0000","last_change":"2024-03-04T15:28:05.500430+0000","last_active":"2024-03-04T15:28:27.719068+0000","last_peered":"2024-03-04T15:28:27.719068+0000","last_clean":"2024-03-04T15:28:27.719068+0000","last_became_active":"2024-03-04T15:28:05.499895+0000","last_became_peered":"2024-03-04T15:28:05.499895+0000","last_unstale":"2024-03-04T15:28:27.719068+0000","last_undegraded":"2024-03-04T15:28:27.719068+0000","last_fullsized":"2024-03-04T15:28:27.719068+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:06:07.421134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718317+0000","last_change":"2024-03-04T15:27:59.468643+0000","last_active":"2024-03-04T15:28:27.718317+0000","last_peered":"2024-03-04T15:28:27.718317+0000","last_clean":"2024-03-04T15:28:27.718317+0000","last_became_active":"2024-03-04T15:27:59.468295+0000","last_became_peered":"2024-03-04T15:27:59.468295+0000","last_unstale":"2024-03-04T15:28:27.718317+0000","last_undegraded":"2024-03-04T15:28:27.718317+0000","last_fullsized":"2024-03-04T15:28:27.718317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:04:59.633181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717040+0000","last_change":"2024-03-04T15:28:01.481755+0000","last_active":"2024-03-04T15:28:27.717040+0000","last_peered":"2024-03-04T15:28:27.717040+0000","last_clean":"2024-03-04T15:28:27.717040+0000","last_became_active":"2024-03-04T15:28:01.481415+0000","last_became_peered":"2024-03-04T15:28:01.481415+0000","last_unstale":"2024-03-04T15:28:27.717040+0000","last_undegraded":"2024-03-04T15:28:27.717040+0000","last_fullsized":"2024-03-04T15:28:27.717040+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:22.852308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717916+0000","last_change":"2024-03-04T15:28:03.482727+0000","last_active":"2024-03-04T15:28:27.717916+0000","last_peered":"2024-03-04T15:28:27.717916+0000","last_clean":"2024-03-04T15:28:27.717916+0000","last_became_active":"2024-03-04T15:28:03.482446+0000","last_became_peered":"2024-03-04T15:28:03.482446+0000","last_unstale":"2024-03-04T15:28:27.717916+0000","last_undegraded":"2024-03-04T15:28:27.717916+0000","last_fullsized":"2024-03-04T15:28:27.717916+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:03:23.013026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716930+0000","last_change":"2024-03-04T15:28:05.502563+0000","last_active":"2024-03-04T15:28:27.716930+0000","last_peered":"2024-03-04T15:28:27.716930+0000","last_clean":"2024-03-04T15:28:27.716930+0000","last_became_active":"2024-03-04T15:28:05.501280+0000","last_became_peered":"2024-03-04T15:28:05.501280+0000","last_unstale":"2024-03-04T15:28:27.716930+0000","last_undegraded":"2024-03-04T15:28:27.716930+0000","last_fullsized":"2024-03-04T15:28:27.716930+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:18:52.239497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513436+0000","last_change":"2024-03-04T15:28:03.484438+0000","last_active":"2024-03-04T15:28:29.513436+0000","last_peered":"2024-03-04T15:28:29.513436+0000","last_clean":"2024-03-04T15:28:29.513436+0000","last_became_active":"2024-03-04T15:28:03.483987+0000","last_became_peered":"2024-03-04T15:28:03.483987+0000","last_unstale":"2024-03-04T15:28:29.513436+0000","last_undegraded":"2024-03-04T15:28:29.513436+0000","last_fullsized":"2024-03-04T15:28:29.513436+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:08:10.496103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715739+0000","last_change":"2024-03-04T15:27:59.467686+0000","last_active":"2024-03-04T15:28:27.715739+0000","last_peered":"2024-03-04T15:28:27.715739+0000","last_clean":"2024-03-04T15:28:27.715739+0000","last_became_active":"2024-03-04T15:27:59.467101+0000","last_became_peered":"2024-03-04T15:27:59.467101+0000","last_unstale":"2024-03-04T15:28:27.715739+0000","last_undegraded":"2024-03-04T15:28:27.715739+0000","last_fullsized":"2024-03-04T15:28:27.715739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:43:33.088437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718608+0000","last_change":"2024-03-04T15:28:01.478029+0000","last_active":"2024-03-04T15:28:27.718608+0000","last_peered":"2024-03-04T15:28:27.718608+0000","last_clean":"2024-03-04T15:28:27.718608+0000","last_became_active":"2024-03-04T15:28:01.477687+0000","last_became_peered":"2024-03-04T15:28:01.477687+0000","last_unstale":"2024-03-04T15:28:27.718608+0000","last_undegraded":"2024-03-04T15:28:27.718608+0000","last_fullsized":"2024-03-04T15:28:27.718608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:39:21.992595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718943+0000","last_change":"2024-03-04T15:28:05.500075+0000","last_active":"2024-03-04T15:28:27.718943+0000","last_peered":"2024-03-04T15:28:27.718943+0000","last_clean":"2024-03-04T15:28:27.718943+0000","last_became_active":"2024-03-04T15:28:05.499647+0000","last_became_peered":"2024-03-04T15:28:05.499647+0000","last_unstale":"2024-03-04T15:28:27.718943+0000","last_undegraded":"2024-03-04T15:28:27.718943+0000","last_fullsized":"2024-03-04T15:28:27.718943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:51:20.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717946+0000","last_change":"2024-03-04T15:28:03.488396+0000","last_active":"2024-03-04T15:28:27.717946+0000","last_peered":"2024-03-04T15:28:27.717946+0000","last_clean":"2024-03-04T15:28:27.717946+0000","last_became_active":"2024-03-04T15:28:03.488095+0000","last_became_peered":"2024-03-04T15:28:03.488095+0000","last_unstale":"2024-03-04T15:28:27.717946+0000","last_undegraded":"2024-03-04T15:28:27.717946+0000","last_fullsized":"2024-03-04T15:28:27.717946+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:25:18.427286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714087+0000","last_change":"2024-03-04T15:27:59.466478+0000","last_active":"2024-03-04T15:28:27.714087+0000","last_peered":"2024-03-04T15:28:27.714087+0000","last_clean":"2024-03-04T15:28:27.714087+0000","last_became_active":"2024-03-04T15:27:59.465601+0000","last_became_peered":"2024-03-04T15:27:59.465601+0000","last_unstale":"2024-03-04T15:28:27.714087+0000","last_undegraded":"2024-03-04T15:28:27.714087+0000","last_fullsized":"2024-03-04T15:28:27.714087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:21:11.816176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718178+0000","last_change":"2024-03-04T15:28:01.477492+0000","last_active":"2024-03-04T15:28:27.718178+0000","last_peered":"2024-03-04T15:28:27.718178+0000","last_clean":"2024-03-04T15:28:27.718178+0000","last_became_active":"2024-03-04T15:28:01.476771+0000","last_became_peered":"2024-03-04T15:28:01.476771+0000","last_unstale":"2024-03-04T15:28:27.718178+0000","last_undegraded":"2024-03-04T15:28:27.718178+0000","last_fullsized":"2024-03-04T15:28:27.718178+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:35:49.229666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720844+0000","last_change":"2024-03-04T15:28:05.500469+0000","last_active":"2024-03-04T15:28:27.720844+0000","last_peered":"2024-03-04T15:28:27.720844+0000","last_clean":"2024-03-04T15:28:27.720844+0000","last_became_active":"2024-03-04T15:28:05.499905+0000","last_became_peered":"2024-03-04T15:28:05.499905+0000","last_unstale":"2024-03-04T15:28:27.720844+0000","last_undegraded":"2024-03-04T15:28:27.720844+0000","last_fullsized":"2024-03-04T15:28:27.720844+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:34:54.428688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719697+0000","last_change":"2024-03-04T15:27:59.464685+0000","last_active":"2024-03-04T15:28:27.719697+0000","last_peered":"2024-03-04T15:28:27.719697+0000","last_clean":"2024-03-04T15:28:27.719697+0000","last_became_active":"2024-03-04T15:27:59.464296+0000","last_became_peered":"2024-03-04T15:27:59.464296+0000","last_unstale":"2024-03-04T15:28:27.719697+0000","last_undegraded":"2024-03-04T15:28:27.719697+0000","last_fullsized":"2024-03-04T15:28:27.719697+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:09:43.453889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714924+0000","last_change":"2024-03-04T15:28:01.478023+0000","last_active":"2024-03-04T15:28:27.714924+0000","last_peered":"2024-03-04T15:28:27.714924+0000","last_clean":"2024-03-04T15:28:27.714924+0000","last_became_active":"2024-03-04T15:28:01.477361+0000","last_became_peered":"2024-03-04T15:28:01.477361+0000","last_unstale":"2024-03-04T15:28:27.714924+0000","last_undegraded":"2024-03-04T15:28:27.714924+0000","last_fullsized":"2024-03-04T15:28:27.714924+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:18:04.146768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717223+0000","last_change":"2024-03-04T15:28:03.486367+0000","last_active":"2024-03-04T15:28:27.717223+0000","last_peered":"2024-03-04T15:28:27.717223+0000","last_clean":"2024-03-04T15:28:27.717223+0000","last_became_active":"2024-03-04T15:28:03.486055+0000","last_became_peered":"2024-03-04T15:28:03.486055+0000","last_unstale":"2024-03-04T15:28:27.717223+0000","last_undegraded":"2024-03-04T15:28:27.717223+0000","last_fullsized":"2024-03-04T15:28:27.717223+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:39:19.640433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716689+0000","last_change":"2024-03-04T15:28:05.499181+0000","last_active":"2024-03-04T15:28:27.716689+0000","last_peered":"2024-03-04T15:28:27.716689+0000","last_clean":"2024-03-04T15:28:27.716689+0000","last_became_active":"2024-03-04T15:28:05.498315+0000","last_became_peered":"2024-03-04T15:28:05.498315+0000","last_unstale":"2024-03-04T15:28:27.716689+0000","last_undegraded":"2024-03-04T15:28:27.716689+0000","last_fullsized":"2024-03-04T15:28:27.716689+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:08.070756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.513399+0000","last_change":"2024-03-04T15:28:03.484301+0000","last_active":"2024-03-04T15:28:29.513399+0000","last_peered":"2024-03-04T15:28:29.513399+0000","last_clean":"2024-03-04T15:28:29.513399+0000","last_became_active":"2024-03-04T15:28:03.483239+0000","last_became_peered":"2024-03-04T15:28:03.483239+0000","last_unstale":"2024-03-04T15:28:29.513399+0000","last_undegraded":"2024-03-04T15:28:29.513399+0000","last_fullsized":"2024-03-04T15:28:29.513399+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:57:55.704394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718744+0000","last_change":"2024-03-04T15:27:59.465486+0000","last_active":"2024-03-04T15:28:27.718744+0000","last_peered":"2024-03-04T15:28:27.718744+0000","last_clean":"2024-03-04T15:28:27.718744+0000","last_became_active":"2024-03-04T15:27:59.465171+0000","last_became_peered":"2024-03-04T15:27:59.465171+0000","last_unstale":"2024-03-04T15:28:27.718744+0000","last_undegraded":"2024-03-04T15:28:27.718744+0000","last_fullsized":"2024-03-04T15:28:27.718744+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:56:41.292712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716536+0000","last_change":"2024-03-04T15:28:01.477758+0000","last_active":"2024-03-04T15:28:27.716536+0000","last_peered":"2024-03-04T15:28:27.716536+0000","last_clean":"2024-03-04T15:28:27.716536+0000","last_became_active":"2024-03-04T15:28:01.477284+0000","last_became_peered":"2024-03-04T15:28:01.477284+0000","last_unstale":"2024-03-04T15:28:27.716536+0000","last_undegraded":"2024-03-04T15:28:27.716536+0000","last_fullsized":"2024-03-04T15:28:27.716536+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:32.810310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718099+0000","last_change":"2024-03-04T15:28:05.500368+0000","last_active":"2024-03-04T15:28:27.718099+0000","last_peered":"2024-03-04T15:28:27.718099+0000","last_clean":"2024-03-04T15:28:27.718099+0000","last_became_active":"2024-03-04T15:28:05.499835+0000","last_became_peered":"2024-03-04T15:28:05.499835+0000","last_unstale":"2024-03-04T15:28:27.718099+0000","last_undegraded":"2024-03-04T15:28:27.718099+0000","last_fullsized":"2024-03-04T15:28:27.718099+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:34.208403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719395+0000","last_change":"2024-03-04T15:27:59.465303+0000","last_active":"2024-03-04T15:28:27.719395+0000","last_peered":"2024-03-04T15:28:27.719395+0000","last_clean":"2024-03-04T15:28:27.719395+0000","last_became_active":"2024-03-04T15:27:59.464756+0000","last_became_peered":"2024-03-04T15:27:59.464756+0000","last_unstale":"2024-03-04T15:28:27.719395+0000","last_undegraded":"2024-03-04T15:28:27.719395+0000","last_fullsized":"2024-03-04T15:28:27.719395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:25:13.372166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718617+0000","last_change":"2024-03-04T15:28:01.477777+0000","last_active":"2024-03-04T15:28:27.718617+0000","last_peered":"2024-03-04T15:28:27.718617+0000","last_clean":"2024-03-04T15:28:27.718617+0000","last_became_active":"2024-03-04T15:28:01.477385+0000","last_became_peered":"2024-03-04T15:28:01.477385+0000","last_unstale":"2024-03-04T15:28:27.718617+0000","last_undegraded":"2024-03-04T15:28:27.718617+0000","last_fullsized":"2024-03-04T15:28:27.718617+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:53:33.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716056+0000","last_change":"2024-03-04T15:28:03.484155+0000","last_active":"2024-03-04T15:28:27.716056+0000","last_peered":"2024-03-04T15:28:27.716056+0000","last_clean":"2024-03-04T15:28:27.716056+0000","last_became_active":"2024-03-04T15:28:03.483252+0000","last_became_peered":"2024-03-04T15:28:03.483252+0000","last_unstale":"2024-03-04T15:28:27.716056+0000","last_undegraded":"2024-03-04T15:28:27.716056+0000","last_fullsized":"2024-03-04T15:28:27.716056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:00:16.640769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713483+0000","last_change":"2024-03-04T15:28:05.497924+0000","last_active":"2024-03-04T15:28:27.713483+0000","last_peered":"2024-03-04T15:28:27.713483+0000","last_clean":"2024-03-04T15:28:27.713483+0000","last_became_active":"2024-03-04T15:28:05.497309+0000","last_became_peered":"2024-03-04T15:28:05.497309+0000","last_unstale":"2024-03-04T15:28:27.713483+0000","last_undegraded":"2024-03-04T15:28:27.713483+0000","last_fullsized":"2024-03-04T15:28:27.713483+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:33:45.755167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720986+0000","last_change":"2024-03-04T15:27:59.468001+0000","last_active":"2024-03-04T15:28:27.720986+0000","last_peered":"2024-03-04T15:28:27.720986+0000","last_clean":"2024-03-04T15:28:27.720986+0000","last_became_active":"2024-03-04T15:27:59.467484+0000","last_became_peered":"2024-03-04T15:27:59.467484+0000","last_unstale":"2024-03-04T15:28:27.720986+0000","last_undegraded":"2024-03-04T15:28:27.720986+0000","last_fullsized":"2024-03-04T15:28:27.720986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:20.983825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717169+0000","last_change":"2024-03-04T15:28:01.478952+0000","last_active":"2024-03-04T15:28:27.717169+0000","last_peered":"2024-03-04T15:28:27.717169+0000","last_clean":"2024-03-04T15:28:27.717169+0000","last_became_active":"2024-03-04T15:28:01.477787+0000","last_became_peered":"2024-03-04T15:28:01.477787+0000","last_unstale":"2024-03-04T15:28:27.717169+0000","last_undegraded":"2024-03-04T15:28:27.717169+0000","last_fullsized":"2024-03-04T15:28:27.717169+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:45:51.535725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714362+0000","last_change":"2024-03-04T15:28:03.483230+0000","last_active":"2024-03-04T15:28:27.714362+0000","last_peered":"2024-03-04T15:28:27.714362+0000","last_clean":"2024-03-04T15:28:27.714362+0000","last_became_active":"2024-03-04T15:28:03.482687+0000","last_became_peered":"2024-03-04T15:28:03.482687+0000","last_unstale":"2024-03-04T15:28:27.714362+0000","last_undegraded":"2024-03-04T15:28:27.714362+0000","last_fullsized":"2024-03-04T15:28:27.714362+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:01.070235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715490+0000","last_change":"2024-03-04T15:28:05.502635+0000","last_active":"2024-03-04T15:28:27.715490+0000","last_peered":"2024-03-04T15:28:27.715490+0000","last_clean":"2024-03-04T15:28:27.715490+0000","last_became_active":"2024-03-04T15:28:05.502164+0000","last_became_peered":"2024-03-04T15:28:05.502164+0000","last_unstale":"2024-03-04T15:28:27.715490+0000","last_undegraded":"2024-03-04T15:28:27.715490+0000","last_fullsized":"2024-03-04T15:28:27.715490+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:05.654891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716950+0000","last_change":"2024-03-04T15:27:59.465157+0000","last_active":"2024-03-04T15:28:27.716950+0000","last_peered":"2024-03-04T15:28:27.716950+0000","last_clean":"2024-03-04T15:28:27.716950+0000","last_became_active":"2024-03-04T15:27:59.464305+0000","last_became_peered":"2024-03-04T15:27:59.464305+0000","last_unstale":"2024-03-04T15:28:27.716950+0000","last_undegraded":"2024-03-04T15:28:27.716950+0000","last_fullsized":"2024-03-04T15:28:27.716950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:29.496585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718379+0000","last_change":"2024-03-04T15:28:01.479269+0000","last_active":"2024-03-04T15:28:27.718379+0000","last_peered":"2024-03-04T15:28:27.718379+0000","last_clean":"2024-03-04T15:28:27.718379+0000","last_became_active":"2024-03-04T15:28:01.478784+0000","last_became_peered":"2024-03-04T15:28:01.478784+0000","last_unstale":"2024-03-04T15:28:27.718379+0000","last_undegraded":"2024-03-04T15:28:27.718379+0000","last_fullsized":"2024-03-04T15:28:27.718379+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:21.598972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718430+0000","last_change":"2024-03-04T15:28:03.483707+0000","last_active":"2024-03-04T15:28:27.718430+0000","last_peered":"2024-03-04T15:28:27.718430+0000","last_clean":"2024-03-04T15:28:27.718430+0000","last_became_active":"2024-03-04T15:28:03.483002+0000","last_became_peered":"2024-03-04T15:28:03.483002+0000","last_unstale":"2024-03-04T15:28:27.718430+0000","last_undegraded":"2024-03-04T15:28:27.718430+0000","last_fullsized":"2024-03-04T15:28:27.718430+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:19:32.170341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411340,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738390,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704688,"statfs":{"total":95990841344,"available":95953600512,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T15:28:41.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T15:28:41.708 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:41.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:41 smithi160 ceph-mon[30717]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T15:28:41.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:41 smithi160 ceph-mon[30717]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:28:41.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:41 smithi064 ceph-mon[30297]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T15:28:41.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:41 smithi064 ceph-mon[30297]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:28:41.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:41 smithi053 ceph-mon[25738]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-04T15:28:41.830 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:41 smithi053 ceph-mon[25738]: from='client.14526 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:28:42.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:42 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ::ffff:172.21.15.64 - - [04/Mar/2024:15:28:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:28:43.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:43 smithi053 ceph-mon[25738]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:28:43.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:43 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:43.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:43 smithi064 ceph-mon[30297]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:28:43.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:43 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:43.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:43 smithi160 ceph-mon[30717]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-04T15:28:43.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:43 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:44.375 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:28:44.377 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-03-04T15:28:45.123 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-04T15:28:43.727335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595910,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":100780,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":281122388,"statfs":{"total":287972524032,"available":287869325312,"internally_reserved":0,"allocated":3158016,"data_stored":2100351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002802"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716024+0000","last_change":"2024-03-04T15:28:05.497122+0000","last_active":"2024-03-04T15:28:27.716024+0000","last_peered":"2024-03-04T15:28:27.716024+0000","last_clean":"2024-03-04T15:28:27.716024+0000","last_became_active":"2024-03-04T15:28:05.496672+0000","last_became_peered":"2024-03-04T15:28:05.496672+0000","last_unstale":"2024-03-04T15:28:27.716024+0000","last_undegraded":"2024-03-04T15:28:27.716024+0000","last_fullsized":"2024-03-04T15:28:27.716024+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:46.473477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717800+0000","last_change":"2024-03-04T15:28:03.479478+0000","last_active":"2024-03-04T15:28:27.717800+0000","last_peered":"2024-03-04T15:28:27.717800+0000","last_clean":"2024-03-04T15:28:27.717800+0000","last_became_active":"2024-03-04T15:28:03.478632+0000","last_became_peered":"2024-03-04T15:28:03.478632+0000","last_unstale":"2024-03-04T15:28:27.717800+0000","last_undegraded":"2024-03-04T15:28:27.717800+0000","last_fullsized":"2024-03-04T15:28:27.717800+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:28:30.174967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718190+0000","last_change":"2024-03-04T15:27:59.464910+0000","last_active":"2024-03-04T15:28:27.718190+0000","last_peered":"2024-03-04T15:28:27.718190+0000","last_clean":"2024-03-04T15:28:27.718190+0000","last_became_active":"2024-03-04T15:27:59.464339+0000","last_became_peered":"2024-03-04T15:27:59.464339+0000","last_unstale":"2024-03-04T15:28:27.718190+0000","last_undegraded":"2024-03-04T15:28:27.718190+0000","last_fullsized":"2024-03-04T15:28:27.718190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:39.262425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716825+0000","last_change":"2024-03-04T15:28:01.478830+0000","last_active":"2024-03-04T15:28:27.716825+0000","last_peered":"2024-03-04T15:28:27.716825+0000","last_clean":"2024-03-04T15:28:27.716825+0000","last_became_active":"2024-03-04T15:28:01.478400+0000","last_became_peered":"2024-03-04T15:28:01.478400+0000","last_unstale":"2024-03-04T15:28:27.716825+0000","last_undegraded":"2024-03-04T15:28:27.716825+0000","last_fullsized":"2024-03-04T15:28:27.716825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:20:31.901452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713982+0000","last_change":"2024-03-04T15:28:05.500883+0000","last_active":"2024-03-04T15:28:27.713982+0000","last_peered":"2024-03-04T15:28:27.713982+0000","last_clean":"2024-03-04T15:28:27.713982+0000","last_became_active":"2024-03-04T15:28:05.500360+0000","last_became_peered":"2024-03-04T15:28:05.500360+0000","last_unstale":"2024-03-04T15:28:27.713982+0000","last_undegraded":"2024-03-04T15:28:27.713982+0000","last_fullsized":"2024-03-04T15:28:27.713982+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:52:22.452537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715852+0000","last_change":"2024-03-04T15:28:03.485974+0000","last_active":"2024-03-04T15:28:27.715852+0000","last_peered":"2024-03-04T15:28:27.715852+0000","last_clean":"2024-03-04T15:28:27.715852+0000","last_became_active":"2024-03-04T15:28:03.485631+0000","last_became_peered":"2024-03-04T15:28:03.485631+0000","last_unstale":"2024-03-04T15:28:27.715852+0000","last_undegraded":"2024-03-04T15:28:27.715852+0000","last_fullsized":"2024-03-04T15:28:27.715852+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:57:10.172676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717224+0000","last_change":"2024-03-04T15:27:59.465221+0000","last_active":"2024-03-04T15:28:27.717224+0000","last_peered":"2024-03-04T15:28:27.717224+0000","last_clean":"2024-03-04T15:28:27.717224+0000","last_became_active":"2024-03-04T15:27:59.464804+0000","last_became_peered":"2024-03-04T15:27:59.464804+0000","last_unstale":"2024-03-04T15:28:27.717224+0000","last_undegraded":"2024-03-04T15:28:27.717224+0000","last_fullsized":"2024-03-04T15:28:27.717224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:47:06.672369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717963+0000","last_change":"2024-03-04T15:28:01.482185+0000","last_active":"2024-03-04T15:28:27.717963+0000","last_peered":"2024-03-04T15:28:27.717963+0000","last_clean":"2024-03-04T15:28:27.717963+0000","last_became_active":"2024-03-04T15:28:01.481517+0000","last_became_peered":"2024-03-04T15:28:01.481517+0000","last_unstale":"2024-03-04T15:28:27.717963+0000","last_undegraded":"2024-03-04T15:28:27.717963+0000","last_fullsized":"2024-03-04T15:28:27.717963+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:21:29.742599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718053+0000","last_change":"2024-03-04T15:28:05.497934+0000","last_active":"2024-03-04T15:28:27.718053+0000","last_peered":"2024-03-04T15:28:27.718053+0000","last_clean":"2024-03-04T15:28:27.718053+0000","last_became_active":"2024-03-04T15:28:05.497649+0000","last_became_peered":"2024-03-04T15:28:05.497649+0000","last_unstale":"2024-03-04T15:28:27.718053+0000","last_undegraded":"2024-03-04T15:28:27.718053+0000","last_fullsized":"2024-03-04T15:28:27.718053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:12:50.364879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716492+0000","last_change":"2024-03-04T15:28:03.486210+0000","last_active":"2024-03-04T15:28:27.716492+0000","last_peered":"2024-03-04T15:28:27.716492+0000","last_clean":"2024-03-04T15:28:27.716492+0000","last_became_active":"2024-03-04T15:28:03.485813+0000","last_became_peered":"2024-03-04T15:28:03.485813+0000","last_unstale":"2024-03-04T15:28:27.716492+0000","last_undegraded":"2024-03-04T15:28:27.716492+0000","last_fullsized":"2024-03-04T15:28:27.716492+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:07.827278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721268+0000","last_change":"2024-03-04T15:27:59.467516+0000","last_active":"2024-03-04T15:28:27.721268+0000","last_peered":"2024-03-04T15:28:27.721268+0000","last_clean":"2024-03-04T15:28:27.721268+0000","last_became_active":"2024-03-04T15:27:59.466680+0000","last_became_peered":"2024-03-04T15:27:59.466680+0000","last_unstale":"2024-03-04T15:28:27.721268+0000","last_undegraded":"2024-03-04T15:28:27.721268+0000","last_fullsized":"2024-03-04T15:28:27.721268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:37:59.565627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717567+0000","last_change":"2024-03-04T15:28:01.477631+0000","last_active":"2024-03-04T15:28:27.717567+0000","last_peered":"2024-03-04T15:28:27.717567+0000","last_clean":"2024-03-04T15:28:27.717567+0000","last_became_active":"2024-03-04T15:28:01.477303+0000","last_became_peered":"2024-03-04T15:28:01.477303+0000","last_unstale":"2024-03-04T15:28:27.717567+0000","last_undegraded":"2024-03-04T15:28:27.717567+0000","last_fullsized":"2024-03-04T15:28:27.717567+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:29:46.954475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716264+0000","last_change":"2024-03-04T15:28:05.501288+0000","last_active":"2024-03-04T15:28:27.716264+0000","last_peered":"2024-03-04T15:28:27.716264+0000","last_clean":"2024-03-04T15:28:27.716264+0000","last_became_active":"2024-03-04T15:28:05.500839+0000","last_became_peered":"2024-03-04T15:28:05.500839+0000","last_unstale":"2024-03-04T15:28:27.716264+0000","last_undegraded":"2024-03-04T15:28:27.716264+0000","last_fullsized":"2024-03-04T15:28:27.716264+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:57:37.966183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715933+0000","last_change":"2024-03-04T15:28:03.485384+0000","last_active":"2024-03-04T15:28:27.715933+0000","last_peered":"2024-03-04T15:28:27.715933+0000","last_clean":"2024-03-04T15:28:27.715933+0000","last_became_active":"2024-03-04T15:28:03.484995+0000","last_became_peered":"2024-03-04T15:28:03.484995+0000","last_unstale":"2024-03-04T15:28:27.715933+0000","last_undegraded":"2024-03-04T15:28:27.715933+0000","last_fullsized":"2024-03-04T15:28:27.715933+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:51:27.872640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718980+0000","last_change":"2024-03-04T15:27:59.465210+0000","last_active":"2024-03-04T15:28:27.718980+0000","last_peered":"2024-03-04T15:28:27.718980+0000","last_clean":"2024-03-04T15:28:27.718980+0000","last_became_active":"2024-03-04T15:27:59.464495+0000","last_became_peered":"2024-03-04T15:27:59.464495+0000","last_unstale":"2024-03-04T15:28:27.718980+0000","last_undegraded":"2024-03-04T15:28:27.718980+0000","last_fullsized":"2024-03-04T15:28:27.718980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:30:02.607815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721188+0000","last_change":"2024-03-04T15:28:01.478147+0000","last_active":"2024-03-04T15:28:27.721188+0000","last_peered":"2024-03-04T15:28:27.721188+0000","last_clean":"2024-03-04T15:28:27.721188+0000","last_became_active":"2024-03-04T15:28:01.477848+0000","last_became_peered":"2024-03-04T15:28:01.477848+0000","last_unstale":"2024-03-04T15:28:27.721188+0000","last_undegraded":"2024-03-04T15:28:27.721188+0000","last_fullsized":"2024-03-04T15:28:27.721188+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:05:53.676778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716620+0000","last_change":"2024-03-04T15:28:05.502385+0000","last_active":"2024-03-04T15:28:27.716620+0000","last_peered":"2024-03-04T15:28:27.716620+0000","last_clean":"2024-03-04T15:28:27.716620+0000","last_became_active":"2024-03-04T15:28:05.501700+0000","last_became_peered":"2024-03-04T15:28:05.501700+0000","last_unstale":"2024-03-04T15:28:27.716620+0000","last_undegraded":"2024-03-04T15:28:27.716620+0000","last_fullsized":"2024-03-04T15:28:27.716620+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:36:24.250295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720879+0000","last_change":"2024-03-04T15:28:03.484195+0000","last_active":"2024-03-04T15:28:27.720879+0000","last_peered":"2024-03-04T15:28:27.720879+0000","last_clean":"2024-03-04T15:28:27.720879+0000","last_became_active":"2024-03-04T15:28:03.483881+0000","last_became_peered":"2024-03-04T15:28:03.483881+0000","last_unstale":"2024-03-04T15:28:27.720879+0000","last_undegraded":"2024-03-04T15:28:27.720879+0000","last_fullsized":"2024-03-04T15:28:27.720879+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:41:07.619198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719622+0000","last_change":"2024-03-04T15:27:59.464506+0000","last_active":"2024-03-04T15:28:27.719622+0000","last_peered":"2024-03-04T15:28:27.719622+0000","last_clean":"2024-03-04T15:28:27.719622+0000","last_became_active":"2024-03-04T15:27:59.464078+0000","last_became_peered":"2024-03-04T15:27:59.464078+0000","last_unstale":"2024-03-04T15:28:27.719622+0000","last_undegraded":"2024-03-04T15:28:27.719622+0000","last_fullsized":"2024-03-04T15:28:27.719622+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:24:46.279408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719794+0000","last_change":"2024-03-04T15:28:01.477571+0000","last_active":"2024-03-04T15:28:27.719794+0000","last_peered":"2024-03-04T15:28:27.719794+0000","last_clean":"2024-03-04T15:28:27.719794+0000","last_became_active":"2024-03-04T15:28:01.477049+0000","last_became_peered":"2024-03-04T15:28:01.477049+0000","last_unstale":"2024-03-04T15:28:27.719794+0000","last_undegraded":"2024-03-04T15:28:27.719794+0000","last_fullsized":"2024-03-04T15:28:27.719794+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:39:52.803286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715969+0000","last_change":"2024-03-04T15:28:05.500221+0000","last_active":"2024-03-04T15:28:27.715969+0000","last_peered":"2024-03-04T15:28:27.715969+0000","last_clean":"2024-03-04T15:28:27.715969+0000","last_became_active":"2024-03-04T15:28:05.499831+0000","last_became_peered":"2024-03-04T15:28:05.499831+0000","last_unstale":"2024-03-04T15:28:27.715969+0000","last_undegraded":"2024-03-04T15:28:27.715969+0000","last_fullsized":"2024-03-04T15:28:27.715969+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:43:12.736111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718766+0000","last_change":"2024-03-04T15:28:03.483624+0000","last_active":"2024-03-04T15:28:27.718766+0000","last_peered":"2024-03-04T15:28:27.718766+0000","last_clean":"2024-03-04T15:28:27.718766+0000","last_became_active":"2024-03-04T15:28:03.483198+0000","last_became_peered":"2024-03-04T15:28:03.483198+0000","last_unstale":"2024-03-04T15:28:27.718766+0000","last_undegraded":"2024-03-04T15:28:27.718766+0000","last_fullsized":"2024-03-04T15:28:27.718766+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:21:43.682326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716193+0000","last_change":"2024-03-04T15:27:59.467016+0000","last_active":"2024-03-04T15:28:27.716193+0000","last_peered":"2024-03-04T15:28:27.716193+0000","last_clean":"2024-03-04T15:28:27.716193+0000","last_became_active":"2024-03-04T15:27:59.466624+0000","last_became_peered":"2024-03-04T15:27:59.466624+0000","last_unstale":"2024-03-04T15:28:27.716193+0000","last_undegraded":"2024-03-04T15:28:27.716193+0000","last_fullsized":"2024-03-04T15:28:27.716193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:37:30.676021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719442+0000","last_change":"2024-03-04T15:28:01.477226+0000","last_active":"2024-03-04T15:28:27.719442+0000","last_peered":"2024-03-04T15:28:27.719442+0000","last_clean":"2024-03-04T15:28:27.719442+0000","last_became_active":"2024-03-04T15:28:01.476893+0000","last_became_peered":"2024-03-04T15:28:01.476893+0000","last_unstale":"2024-03-04T15:28:27.719442+0000","last_undegraded":"2024-03-04T15:28:27.719442+0000","last_fullsized":"2024-03-04T15:28:27.719442+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:18:36.974992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716415+0000","last_change":"2024-03-04T15:28:05.501291+0000","last_active":"2024-03-04T15:28:27.716415+0000","last_peered":"2024-03-04T15:28:27.716415+0000","last_clean":"2024-03-04T15:28:27.716415+0000","last_became_active":"2024-03-04T15:28:05.500329+0000","last_became_peered":"2024-03-04T15:28:05.500329+0000","last_unstale":"2024-03-04T15:28:27.716415+0000","last_undegraded":"2024-03-04T15:28:27.716415+0000","last_fullsized":"2024-03-04T15:28:27.716415+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:19:15.027929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513870+0000","last_change":"2024-03-04T15:28:03.483892+0000","last_active":"2024-03-04T15:28:34.513870+0000","last_peered":"2024-03-04T15:28:34.513870+0000","last_clean":"2024-03-04T15:28:34.513870+0000","last_became_active":"2024-03-04T15:28:03.482840+0000","last_became_peered":"2024-03-04T15:28:03.482840+0000","last_unstale":"2024-03-04T15:28:34.513870+0000","last_undegraded":"2024-03-04T15:28:34.513870+0000","last_fullsized":"2024-03-04T15:28:34.513870+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:37:48.021076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714196+0000","last_change":"2024-03-04T15:27:59.466120+0000","last_active":"2024-03-04T15:28:27.714196+0000","last_peered":"2024-03-04T15:28:27.714196+0000","last_clean":"2024-03-04T15:28:27.714196+0000","last_became_active":"2024-03-04T15:27:59.463786+0000","last_became_peered":"2024-03-04T15:27:59.463786+0000","last_unstale":"2024-03-04T15:28:27.714196+0000","last_undegraded":"2024-03-04T15:28:27.714196+0000","last_fullsized":"2024-03-04T15:28:27.714196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:53:11.301306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713482+0000","last_change":"2024-03-04T15:28:01.478331+0000","last_active":"2024-03-04T15:28:27.713482+0000","last_peered":"2024-03-04T15:28:27.713482+0000","last_clean":"2024-03-04T15:28:27.713482+0000","last_became_active":"2024-03-04T15:28:01.477947+0000","last_became_peered":"2024-03-04T15:28:01.477947+0000","last_unstale":"2024-03-04T15:28:27.713482+0000","last_undegraded":"2024-03-04T15:28:27.713482+0000","last_fullsized":"2024-03-04T15:28:27.713482+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:48:40.390358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716810+0000","last_change":"2024-03-04T15:28:05.503387+0000","last_active":"2024-03-04T15:28:27.716810+0000","last_peered":"2024-03-04T15:28:27.716810+0000","last_clean":"2024-03-04T15:28:27.716810+0000","last_became_active":"2024-03-04T15:28:05.502717+0000","last_became_peered":"2024-03-04T15:28:05.502717+0000","last_unstale":"2024-03-04T15:28:27.716810+0000","last_undegraded":"2024-03-04T15:28:27.716810+0000","last_fullsized":"2024-03-04T15:28:27.716810+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:26:00.647170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716575+0000","last_change":"2024-03-04T15:28:03.483366+0000","last_active":"2024-03-04T15:28:27.716575+0000","last_peered":"2024-03-04T15:28:27.716575+0000","last_clean":"2024-03-04T15:28:27.716575+0000","last_became_active":"2024-03-04T15:28:03.482751+0000","last_became_peered":"2024-03-04T15:28:03.482751+0000","last_unstale":"2024-03-04T15:28:27.716575+0000","last_undegraded":"2024-03-04T15:28:27.716575+0000","last_fullsized":"2024-03-04T15:28:27.716575+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:17:53.826743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717916+0000","last_change":"2024-03-04T15:27:59.465916+0000","last_active":"2024-03-04T15:28:27.717916+0000","last_peered":"2024-03-04T15:28:27.717916+0000","last_clean":"2024-03-04T15:28:27.717916+0000","last_became_active":"2024-03-04T15:27:59.465557+0000","last_became_peered":"2024-03-04T15:27:59.465557+0000","last_unstale":"2024-03-04T15:28:27.717916+0000","last_undegraded":"2024-03-04T15:28:27.717916+0000","last_fullsized":"2024-03-04T15:28:27.717916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:51:25.628723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713764+0000","last_change":"2024-03-04T15:28:01.475943+0000","last_active":"2024-03-04T15:28:27.713764+0000","last_peered":"2024-03-04T15:28:27.713764+0000","last_clean":"2024-03-04T15:28:27.713764+0000","last_became_active":"2024-03-04T15:28:01.475596+0000","last_became_peered":"2024-03-04T15:28:01.475596+0000","last_unstale":"2024-03-04T15:28:27.713764+0000","last_undegraded":"2024-03-04T15:28:27.713764+0000","last_fullsized":"2024-03-04T15:28:27.713764+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:50:21.697880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716139+0000","last_change":"2024-03-04T15:28:05.500105+0000","last_active":"2024-03-04T15:28:27.716139+0000","last_peered":"2024-03-04T15:28:27.716139+0000","last_clean":"2024-03-04T15:28:27.716139+0000","last_became_active":"2024-03-04T15:28:05.499740+0000","last_became_peered":"2024-03-04T15:28:05.499740+0000","last_unstale":"2024-03-04T15:28:27.716139+0000","last_undegraded":"2024-03-04T15:28:27.716139+0000","last_fullsized":"2024-03-04T15:28:27.716139+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:44:25.099992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713598+0000","last_change":"2024-03-04T15:28:03.483182+0000","last_active":"2024-03-04T15:28:27.713598+0000","last_peered":"2024-03-04T15:28:27.713598+0000","last_clean":"2024-03-04T15:28:27.713598+0000","last_became_active":"2024-03-04T15:28:03.482745+0000","last_became_peered":"2024-03-04T15:28:03.482745+0000","last_unstale":"2024-03-04T15:28:27.713598+0000","last_undegraded":"2024-03-04T15:28:27.713598+0000","last_fullsized":"2024-03-04T15:28:27.713598+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:35.644553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718552+0000","last_change":"2024-03-04T15:27:59.468542+0000","last_active":"2024-03-04T15:28:27.718552+0000","last_peered":"2024-03-04T15:28:27.718552+0000","last_clean":"2024-03-04T15:28:27.718552+0000","last_became_active":"2024-03-04T15:27:59.468144+0000","last_became_peered":"2024-03-04T15:27:59.468144+0000","last_unstale":"2024-03-04T15:28:27.718552+0000","last_undegraded":"2024-03-04T15:28:27.718552+0000","last_fullsized":"2024-03-04T15:28:27.718552+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:28:31.313829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717582+0000","last_change":"2024-03-04T15:28:01.479383+0000","last_active":"2024-03-04T15:28:27.717582+0000","last_peered":"2024-03-04T15:28:27.717582+0000","last_clean":"2024-03-04T15:28:27.717582+0000","last_became_active":"2024-03-04T15:28:01.478835+0000","last_became_peered":"2024-03-04T15:28:01.478835+0000","last_unstale":"2024-03-04T15:28:27.717582+0000","last_undegraded":"2024-03-04T15:28:27.717582+0000","last_fullsized":"2024-03-04T15:28:27.717582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:54:31.667939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716122+0000","last_change":"2024-03-04T15:28:05.502051+0000","last_active":"2024-03-04T15:28:27.716122+0000","last_peered":"2024-03-04T15:28:27.716122+0000","last_clean":"2024-03-04T15:28:27.716122+0000","last_became_active":"2024-03-04T15:28:05.500240+0000","last_became_peered":"2024-03-04T15:28:05.500240+0000","last_unstale":"2024-03-04T15:28:27.716122+0000","last_undegraded":"2024-03-04T15:28:27.716122+0000","last_fullsized":"2024-03-04T15:28:27.716122+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:55:26.833913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713966+0000","last_change":"2024-03-04T15:28:03.483169+0000","last_active":"2024-03-04T15:28:27.713966+0000","last_peered":"2024-03-04T15:28:27.713966+0000","last_clean":"2024-03-04T15:28:27.713966+0000","last_became_active":"2024-03-04T15:28:03.482660+0000","last_became_peered":"2024-03-04T15:28:03.482660+0000","last_unstale":"2024-03-04T15:28:27.713966+0000","last_undegraded":"2024-03-04T15:28:27.713966+0000","last_fullsized":"2024-03-04T15:28:27.713966+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:15:47.790176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717326+0000","last_change":"2024-03-04T15:27:59.469994+0000","last_active":"2024-03-04T15:28:27.717326+0000","last_peered":"2024-03-04T15:28:27.717326+0000","last_clean":"2024-03-04T15:28:27.717326+0000","last_became_active":"2024-03-04T15:27:59.469245+0000","last_became_peered":"2024-03-04T15:27:59.469245+0000","last_unstale":"2024-03-04T15:28:27.717326+0000","last_undegraded":"2024-03-04T15:28:27.717326+0000","last_fullsized":"2024-03-04T15:28:27.717326+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:53:34.199893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718044+0000","last_change":"2024-03-04T15:28:01.481892+0000","last_active":"2024-03-04T15:28:27.718044+0000","last_peered":"2024-03-04T15:28:27.718044+0000","last_clean":"2024-03-04T15:28:27.718044+0000","last_became_active":"2024-03-04T15:28:01.481508+0000","last_became_peered":"2024-03-04T15:28:01.481508+0000","last_unstale":"2024-03-04T15:28:27.718044+0000","last_undegraded":"2024-03-04T15:28:27.718044+0000","last_fullsized":"2024-03-04T15:28:27.718044+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:49:17.811926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719023+0000","last_change":"2024-03-04T15:28:05.498049+0000","last_active":"2024-03-04T15:28:27.719023+0000","last_peered":"2024-03-04T15:28:27.719023+0000","last_clean":"2024-03-04T15:28:27.719023+0000","last_became_active":"2024-03-04T15:28:05.497801+0000","last_became_peered":"2024-03-04T15:28:05.497801+0000","last_unstale":"2024-03-04T15:28:27.719023+0000","last_undegraded":"2024-03-04T15:28:27.719023+0000","last_fullsized":"2024-03-04T15:28:27.719023+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:03:55.560117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716918+0000","last_change":"2024-03-04T15:28:03.484029+0000","last_active":"2024-03-04T15:28:27.716918+0000","last_peered":"2024-03-04T15:28:27.716918+0000","last_clean":"2024-03-04T15:28:27.716918+0000","last_became_active":"2024-03-04T15:28:03.482746+0000","last_became_peered":"2024-03-04T15:28:03.482746+0000","last_unstale":"2024-03-04T15:28:27.716918+0000","last_undegraded":"2024-03-04T15:28:27.716918+0000","last_fullsized":"2024-03-04T15:28:27.716918+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:06:58.224350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721183+0000","last_change":"2024-03-04T15:27:59.467797+0000","last_active":"2024-03-04T15:28:27.721183+0000","last_peered":"2024-03-04T15:28:27.721183+0000","last_clean":"2024-03-04T15:28:27.721183+0000","last_became_active":"2024-03-04T15:27:59.466906+0000","last_became_peered":"2024-03-04T15:27:59.466906+0000","last_unstale":"2024-03-04T15:28:27.721183+0000","last_undegraded":"2024-03-04T15:28:27.721183+0000","last_fullsized":"2024-03-04T15:28:27.721183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:14:38.401121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717418+0000","last_change":"2024-03-04T15:28:01.479770+0000","last_active":"2024-03-04T15:28:27.717418+0000","last_peered":"2024-03-04T15:28:27.717418+0000","last_clean":"2024-03-04T15:28:27.717418+0000","last_became_active":"2024-03-04T15:28:01.479195+0000","last_became_peered":"2024-03-04T15:28:01.479195+0000","last_unstale":"2024-03-04T15:28:27.717418+0000","last_undegraded":"2024-03-04T15:28:27.717418+0000","last_fullsized":"2024-03-04T15:28:27.717418+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:02:49.179079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720368+0000","last_change":"2024-03-04T15:28:05.497142+0000","last_active":"2024-03-04T15:28:27.720368+0000","last_peered":"2024-03-04T15:28:27.720368+0000","last_clean":"2024-03-04T15:28:27.720368+0000","last_became_active":"2024-03-04T15:28:05.496176+0000","last_became_peered":"2024-03-04T15:28:05.496176+0000","last_unstale":"2024-03-04T15:28:27.720368+0000","last_undegraded":"2024-03-04T15:28:27.720368+0000","last_fullsized":"2024-03-04T15:28:27.720368+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:44:55.290706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713752+0000","last_change":"2024-03-04T15:28:03.482722+0000","last_active":"2024-03-04T15:28:27.713752+0000","last_peered":"2024-03-04T15:28:27.713752+0000","last_clean":"2024-03-04T15:28:27.713752+0000","last_became_active":"2024-03-04T15:28:03.482340+0000","last_became_peered":"2024-03-04T15:28:03.482340+0000","last_unstale":"2024-03-04T15:28:27.713752+0000","last_undegraded":"2024-03-04T15:28:27.713752+0000","last_fullsized":"2024-03-04T15:28:27.713752+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:25:43.423784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717074+0000","last_change":"2024-03-04T15:27:59.467849+0000","last_active":"2024-03-04T15:28:27.717074+0000","last_peered":"2024-03-04T15:28:27.717074+0000","last_clean":"2024-03-04T15:28:27.717074+0000","last_became_active":"2024-03-04T15:27:59.466835+0000","last_became_peered":"2024-03-04T15:27:59.466835+0000","last_unstale":"2024-03-04T15:28:27.717074+0000","last_undegraded":"2024-03-04T15:28:27.717074+0000","last_fullsized":"2024-03-04T15:28:27.717074+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:36:43.023274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721034+0000","last_change":"2024-03-04T15:28:01.477243+0000","last_active":"2024-03-04T15:28:27.721034+0000","last_peered":"2024-03-04T15:28:27.721034+0000","last_clean":"2024-03-04T15:28:27.721034+0000","last_became_active":"2024-03-04T15:28:01.476903+0000","last_became_peered":"2024-03-04T15:28:01.476903+0000","last_unstale":"2024-03-04T15:28:27.721034+0000","last_undegraded":"2024-03-04T15:28:27.721034+0000","last_fullsized":"2024-03-04T15:28:27.721034+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:12:26.329137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716983+0000","last_change":"2024-03-04T15:28:05.500635+0000","last_active":"2024-03-04T15:28:27.716983+0000","last_peered":"2024-03-04T15:28:27.716983+0000","last_clean":"2024-03-04T15:28:27.716983+0000","last_became_active":"2024-03-04T15:28:05.500042+0000","last_became_peered":"2024-03-04T15:28:05.500042+0000","last_unstale":"2024-03-04T15:28:27.716983+0000","last_undegraded":"2024-03-04T15:28:27.716983+0000","last_fullsized":"2024-03-04T15:28:27.716983+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:30:30.938794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513843+0000","last_change":"2024-03-04T15:28:03.483434+0000","last_active":"2024-03-04T15:28:34.513843+0000","last_peered":"2024-03-04T15:28:34.513843+0000","last_clean":"2024-03-04T15:28:34.513843+0000","last_became_active":"2024-03-04T15:28:03.482734+0000","last_became_peered":"2024-03-04T15:28:03.482734+0000","last_unstale":"2024-03-04T15:28:34.513843+0000","last_undegraded":"2024-03-04T15:28:34.513843+0000","last_fullsized":"2024-03-04T15:28:34.513843+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:29:32.455559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718992+0000","last_change":"2024-03-04T15:27:59.464412+0000","last_active":"2024-03-04T15:28:27.718992+0000","last_peered":"2024-03-04T15:28:27.718992+0000","last_clean":"2024-03-04T15:28:27.718992+0000","last_became_active":"2024-03-04T15:27:59.464049+0000","last_became_peered":"2024-03-04T15:27:59.464049+0000","last_unstale":"2024-03-04T15:28:27.718992+0000","last_undegraded":"2024-03-04T15:28:27.718992+0000","last_fullsized":"2024-03-04T15:28:27.718992+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:48:26.542755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715467+0000","last_change":"2024-03-04T15:28:01.477596+0000","last_active":"2024-03-04T15:28:27.715467+0000","last_peered":"2024-03-04T15:28:27.715467+0000","last_clean":"2024-03-04T15:28:27.715467+0000","last_became_active":"2024-03-04T15:28:01.477080+0000","last_became_peered":"2024-03-04T15:28:01.477080+0000","last_unstale":"2024-03-04T15:28:27.715467+0000","last_undegraded":"2024-03-04T15:28:27.715467+0000","last_fullsized":"2024-03-04T15:28:27.715467+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:38:26.075233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717301+0000","last_change":"2024-03-04T15:28:05.502039+0000","last_active":"2024-03-04T15:28:27.717301+0000","last_peered":"2024-03-04T15:28:27.717301+0000","last_clean":"2024-03-04T15:28:27.717301+0000","last_became_active":"2024-03-04T15:28:05.501453+0000","last_became_peered":"2024-03-04T15:28:05.501453+0000","last_unstale":"2024-03-04T15:28:27.717301+0000","last_undegraded":"2024-03-04T15:28:27.717301+0000","last_fullsized":"2024-03-04T15:28:27.717301+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:00:43.927613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513802+0000","last_change":"2024-03-04T15:28:03.483909+0000","last_active":"2024-03-04T15:28:34.513802+0000","last_peered":"2024-03-04T15:28:34.513802+0000","last_clean":"2024-03-04T15:28:34.513802+0000","last_became_active":"2024-03-04T15:28:03.483040+0000","last_became_peered":"2024-03-04T15:28:03.483040+0000","last_unstale":"2024-03-04T15:28:34.513802+0000","last_undegraded":"2024-03-04T15:28:34.513802+0000","last_fullsized":"2024-03-04T15:28:34.513802+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:18:24.067938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719383+0000","last_change":"2024-03-04T15:27:59.465444+0000","last_active":"2024-03-04T15:28:27.719383+0000","last_peered":"2024-03-04T15:28:27.719383+0000","last_clean":"2024-03-04T15:28:27.719383+0000","last_became_active":"2024-03-04T15:27:59.464144+0000","last_became_peered":"2024-03-04T15:27:59.464144+0000","last_unstale":"2024-03-04T15:28:27.719383+0000","last_undegraded":"2024-03-04T15:28:27.719383+0000","last_fullsized":"2024-03-04T15:28:27.719383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:02:11.715204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717406+0000","last_change":"2024-03-04T15:28:01.477055+0000","last_active":"2024-03-04T15:28:27.717406+0000","last_peered":"2024-03-04T15:28:27.717406+0000","last_clean":"2024-03-04T15:28:27.717406+0000","last_became_active":"2024-03-04T15:28:01.475829+0000","last_became_peered":"2024-03-04T15:28:01.475829+0000","last_unstale":"2024-03-04T15:28:27.717406+0000","last_undegraded":"2024-03-04T15:28:27.717406+0000","last_fullsized":"2024-03-04T15:28:27.717406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:49:43.088367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716859+0000","last_change":"2024-03-04T15:28:05.502063+0000","last_active":"2024-03-04T15:28:27.716859+0000","last_peered":"2024-03-04T15:28:27.716859+0000","last_clean":"2024-03-04T15:28:27.716859+0000","last_became_active":"2024-03-04T15:28:05.501410+0000","last_became_peered":"2024-03-04T15:28:05.501410+0000","last_unstale":"2024-03-04T15:28:27.716859+0000","last_undegraded":"2024-03-04T15:28:27.716859+0000","last_fullsized":"2024-03-04T15:28:27.716859+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:13:32.805593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717896+0000","last_change":"2024-03-04T15:28:03.487646+0000","last_active":"2024-03-04T15:28:27.717896+0000","last_peered":"2024-03-04T15:28:27.717896+0000","last_clean":"2024-03-04T15:28:27.717896+0000","last_became_active":"2024-03-04T15:28:03.487244+0000","last_became_peered":"2024-03-04T15:28:03.487244+0000","last_unstale":"2024-03-04T15:28:27.717896+0000","last_undegraded":"2024-03-04T15:28:27.717896+0000","last_fullsized":"2024-03-04T15:28:27.717896+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:38:33.760668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715865+0000","last_change":"2024-03-04T15:27:59.467335+0000","last_active":"2024-03-04T15:28:27.715865+0000","last_peered":"2024-03-04T15:28:27.715865+0000","last_clean":"2024-03-04T15:28:27.715865+0000","last_became_active":"2024-03-04T15:27:59.466401+0000","last_became_peered":"2024-03-04T15:27:59.466401+0000","last_unstale":"2024-03-04T15:28:27.715865+0000","last_undegraded":"2024-03-04T15:28:27.715865+0000","last_fullsized":"2024-03-04T15:28:27.715865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:17:13.079130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715494+0000","last_change":"2024-03-04T15:28:01.476066+0000","last_active":"2024-03-04T15:28:27.715494+0000","last_peered":"2024-03-04T15:28:27.715494+0000","last_clean":"2024-03-04T15:28:27.715494+0000","last_became_active":"2024-03-04T15:28:01.475619+0000","last_became_peered":"2024-03-04T15:28:01.475619+0000","last_unstale":"2024-03-04T15:28:27.715494+0000","last_undegraded":"2024-03-04T15:28:27.715494+0000","last_fullsized":"2024-03-04T15:28:27.715494+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:56:51.312084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717673+0000","last_change":"2024-03-04T15:28:05.501493+0000","last_active":"2024-03-04T15:28:27.717673+0000","last_peered":"2024-03-04T15:28:27.717673+0000","last_clean":"2024-03-04T15:28:27.717673+0000","last_became_active":"2024-03-04T15:28:05.501135+0000","last_became_peered":"2024-03-04T15:28:05.501135+0000","last_unstale":"2024-03-04T15:28:27.717673+0000","last_undegraded":"2024-03-04T15:28:27.717673+0000","last_fullsized":"2024-03-04T15:28:27.717673+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:31:33.148825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716663+0000","last_change":"2024-03-04T15:28:03.487580+0000","last_active":"2024-03-04T15:28:27.716663+0000","last_peered":"2024-03-04T15:28:27.716663+0000","last_clean":"2024-03-04T15:28:27.716663+0000","last_became_active":"2024-03-04T15:28:03.487083+0000","last_became_peered":"2024-03-04T15:28:03.487083+0000","last_unstale":"2024-03-04T15:28:27.716663+0000","last_undegraded":"2024-03-04T15:28:27.716663+0000","last_fullsized":"2024-03-04T15:28:27.716663+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T03:07:22.311701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719705+0000","last_change":"2024-03-04T15:27:59.465840+0000","last_active":"2024-03-04T15:28:27.719705+0000","last_peered":"2024-03-04T15:28:27.719705+0000","last_clean":"2024-03-04T15:28:27.719705+0000","last_became_active":"2024-03-04T15:27:59.465362+0000","last_became_peered":"2024-03-04T15:27:59.465362+0000","last_unstale":"2024-03-04T15:28:27.719705+0000","last_undegraded":"2024-03-04T15:28:27.719705+0000","last_fullsized":"2024-03-04T15:28:27.719705+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:25.791581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718251+0000","last_change":"2024-03-04T15:28:01.477714+0000","last_active":"2024-03-04T15:28:27.718251+0000","last_peered":"2024-03-04T15:28:27.718251+0000","last_clean":"2024-03-04T15:28:27.718251+0000","last_became_active":"2024-03-04T15:28:01.477273+0000","last_became_peered":"2024-03-04T15:28:01.477273+0000","last_unstale":"2024-03-04T15:28:27.718251+0000","last_undegraded":"2024-03-04T15:28:27.718251+0000","last_fullsized":"2024-03-04T15:28:27.718251+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:24:40.858313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717636+0000","last_change":"2024-03-04T15:28:05.497141+0000","last_active":"2024-03-04T15:28:27.717636+0000","last_peered":"2024-03-04T15:28:27.717636+0000","last_clean":"2024-03-04T15:28:27.717636+0000","last_became_active":"2024-03-04T15:28:05.496285+0000","last_became_peered":"2024-03-04T15:28:05.496285+0000","last_unstale":"2024-03-04T15:28:27.717636+0000","last_undegraded":"2024-03-04T15:28:27.717636+0000","last_fullsized":"2024-03-04T15:28:27.717636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:48:06.238963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'3","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720621+0000","last_change":"2024-03-04T15:27:59.467227+0000","last_active":"2024-03-04T15:28:27.720621+0000","last_peered":"2024-03-04T15:28:27.720621+0000","last_clean":"2024-03-04T15:28:27.720621+0000","last_became_active":"2024-03-04T15:27:59.466669+0000","last_became_peered":"2024-03-04T15:27:59.466669+0000","last_unstale":"2024-03-04T15:28:27.720621+0000","last_undegraded":"2024-03-04T15:28:27.720621+0000","last_fullsized":"2024-03-04T15:28:27.720621+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:23:57.085399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":490,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719407+0000","last_change":"2024-03-04T15:28:01.478022+0000","last_active":"2024-03-04T15:28:27.719407+0000","last_peered":"2024-03-04T15:28:27.719407+0000","last_clean":"2024-03-04T15:28:27.719407+0000","last_became_active":"2024-03-04T15:28:01.477701+0000","last_became_peered":"2024-03-04T15:28:01.477701+0000","last_unstale":"2024-03-04T15:28:27.719407+0000","last_undegraded":"2024-03-04T15:28:27.719407+0000","last_fullsized":"2024-03-04T15:28:27.719407+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:03:19.988069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513818+0000","last_change":"2024-03-04T15:28:03.483828+0000","last_active":"2024-03-04T15:28:34.513818+0000","last_peered":"2024-03-04T15:28:34.513818+0000","last_clean":"2024-03-04T15:28:34.513818+0000","last_became_active":"2024-03-04T15:28:03.483535+0000","last_became_peered":"2024-03-04T15:28:03.483535+0000","last_unstale":"2024-03-04T15:28:34.513818+0000","last_undegraded":"2024-03-04T15:28:34.513818+0000","last_fullsized":"2024-03-04T15:28:34.513818+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:06:34.065643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714234+0000","last_change":"2024-03-04T15:28:05.497753+0000","last_active":"2024-03-04T15:28:27.714234+0000","last_peered":"2024-03-04T15:28:27.714234+0000","last_clean":"2024-03-04T15:28:27.714234+0000","last_became_active":"2024-03-04T15:28:05.497343+0000","last_became_peered":"2024-03-04T15:28:05.497343+0000","last_unstale":"2024-03-04T15:28:27.714234+0000","last_undegraded":"2024-03-04T15:28:27.714234+0000","last_fullsized":"2024-03-04T15:28:27.714234+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:57:30.610429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717291+0000","last_change":"2024-03-04T15:27:59.464692+0000","last_active":"2024-03-04T15:28:27.717291+0000","last_peered":"2024-03-04T15:28:27.717291+0000","last_clean":"2024-03-04T15:28:27.717291+0000","last_became_active":"2024-03-04T15:27:59.463639+0000","last_became_peered":"2024-03-04T15:27:59.463639+0000","last_unstale":"2024-03-04T15:28:27.717291+0000","last_undegraded":"2024-03-04T15:28:27.717291+0000","last_fullsized":"2024-03-04T15:28:27.717291+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:54:55.898856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513770+0000","last_change":"2024-03-04T15:28:01.476694+0000","last_active":"2024-03-04T15:28:34.513770+0000","last_peered":"2024-03-04T15:28:34.513770+0000","last_clean":"2024-03-04T15:28:34.513770+0000","last_became_active":"2024-03-04T15:28:01.475719+0000","last_became_peered":"2024-03-04T15:28:01.475719+0000","last_unstale":"2024-03-04T15:28:34.513770+0000","last_undegraded":"2024-03-04T15:28:34.513770+0000","last_fullsized":"2024-03-04T15:28:34.513770+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:42:00.331170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719144+0000","last_change":"2024-03-04T15:28:03.483112+0000","last_active":"2024-03-04T15:28:27.719144+0000","last_peered":"2024-03-04T15:28:27.719144+0000","last_clean":"2024-03-04T15:28:27.719144+0000","last_became_active":"2024-03-04T15:28:03.482821+0000","last_became_peered":"2024-03-04T15:28:03.482821+0000","last_unstale":"2024-03-04T15:28:27.719144+0000","last_undegraded":"2024-03-04T15:28:27.719144+0000","last_fullsized":"2024-03-04T15:28:27.719144+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:57:19.759092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715448+0000","last_change":"2024-03-04T15:28:05.498126+0000","last_active":"2024-03-04T15:28:27.715448+0000","last_peered":"2024-03-04T15:28:27.715448+0000","last_clean":"2024-03-04T15:28:27.715448+0000","last_became_active":"2024-03-04T15:28:05.497577+0000","last_became_peered":"2024-03-04T15:28:05.497577+0000","last_unstale":"2024-03-04T15:28:27.715448+0000","last_undegraded":"2024-03-04T15:28:27.715448+0000","last_fullsized":"2024-03-04T15:28:27.715448+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:23:24.187045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716375+0000","last_change":"2024-03-04T15:27:59.469243+0000","last_active":"2024-03-04T15:28:27.716375+0000","last_peered":"2024-03-04T15:28:27.716375+0000","last_clean":"2024-03-04T15:28:27.716375+0000","last_became_active":"2024-03-04T15:27:59.468655+0000","last_became_peered":"2024-03-04T15:27:59.468655+0000","last_unstale":"2024-03-04T15:28:27.716375+0000","last_undegraded":"2024-03-04T15:28:27.716375+0000","last_fullsized":"2024-03-04T15:28:27.716375+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:50:13.408312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717533+0000","last_change":"2024-03-04T15:28:01.477060+0000","last_active":"2024-03-04T15:28:27.717533+0000","last_peered":"2024-03-04T15:28:27.717533+0000","last_clean":"2024-03-04T15:28:27.717533+0000","last_became_active":"2024-03-04T15:28:01.476214+0000","last_became_peered":"2024-03-04T15:28:01.476214+0000","last_unstale":"2024-03-04T15:28:27.717533+0000","last_undegraded":"2024-03-04T15:28:27.717533+0000","last_fullsized":"2024-03-04T15:28:27.717533+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:52:45.695981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717075+0000","last_change":"2024-03-04T15:28:03.483233+0000","last_active":"2024-03-04T15:28:27.717075+0000","last_peered":"2024-03-04T15:28:27.717075+0000","last_clean":"2024-03-04T15:28:27.717075+0000","last_became_active":"2024-03-04T15:28:03.482737+0000","last_became_peered":"2024-03-04T15:28:03.482737+0000","last_unstale":"2024-03-04T15:28:27.717075+0000","last_undegraded":"2024-03-04T15:28:27.717075+0000","last_fullsized":"2024-03-04T15:28:27.717075+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:53.633584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717473+0000","last_change":"2024-03-04T15:28:05.501456+0000","last_active":"2024-03-04T15:28:27.717473+0000","last_peered":"2024-03-04T15:28:27.717473+0000","last_clean":"2024-03-04T15:28:27.717473+0000","last_became_active":"2024-03-04T15:28:05.501099+0000","last_became_peered":"2024-03-04T15:28:05.501099+0000","last_unstale":"2024-03-04T15:28:27.717473+0000","last_undegraded":"2024-03-04T15:28:27.717473+0000","last_fullsized":"2024-03-04T15:28:27.717473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:53:43.295522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717727+0000","last_change":"2024-03-04T15:28:01.478401+0000","last_active":"2024-03-04T15:28:27.717727+0000","last_peered":"2024-03-04T15:28:27.717727+0000","last_clean":"2024-03-04T15:28:27.717727+0000","last_became_active":"2024-03-04T15:28:01.478084+0000","last_became_peered":"2024-03-04T15:28:01.478084+0000","last_unstale":"2024-03-04T15:28:27.717727+0000","last_undegraded":"2024-03-04T15:28:27.717727+0000","last_fullsized":"2024-03-04T15:28:27.717727+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:21:42.662935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713947+0000","last_change":"2024-03-04T15:27:59.466334+0000","last_active":"2024-03-04T15:28:27.713947+0000","last_peered":"2024-03-04T15:28:27.713947+0000","last_clean":"2024-03-04T15:28:27.713947+0000","last_became_active":"2024-03-04T15:27:59.465459+0000","last_became_peered":"2024-03-04T15:27:59.465459+0000","last_unstale":"2024-03-04T15:28:27.713947+0000","last_undegraded":"2024-03-04T15:28:27.713947+0000","last_fullsized":"2024-03-04T15:28:27.713947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:01:52.266865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718033+0000","last_change":"2024-03-04T15:28:03.488694+0000","last_active":"2024-03-04T15:28:27.718033+0000","last_peered":"2024-03-04T15:28:27.718033+0000","last_clean":"2024-03-04T15:28:27.718033+0000","last_became_active":"2024-03-04T15:28:03.488408+0000","last_became_peered":"2024-03-04T15:28:03.488408+0000","last_unstale":"2024-03-04T15:28:27.718033+0000","last_undegraded":"2024-03-04T15:28:27.718033+0000","last_fullsized":"2024-03-04T15:28:27.718033+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:17:47.930500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718217+0000","last_change":"2024-03-04T15:28:05.500191+0000","last_active":"2024-03-04T15:28:27.718217+0000","last_peered":"2024-03-04T15:28:27.718217+0000","last_clean":"2024-03-04T15:28:27.718217+0000","last_became_active":"2024-03-04T15:28:05.499715+0000","last_became_peered":"2024-03-04T15:28:05.499715+0000","last_unstale":"2024-03-04T15:28:27.718217+0000","last_undegraded":"2024-03-04T15:28:27.718217+0000","last_fullsized":"2024-03-04T15:28:27.718217+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:25:14.932681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715166+0000","last_change":"2024-03-04T15:28:01.475816+0000","last_active":"2024-03-04T15:28:27.715166+0000","last_peered":"2024-03-04T15:28:27.715166+0000","last_clean":"2024-03-04T15:28:27.715166+0000","last_became_active":"2024-03-04T15:28:01.475240+0000","last_became_peered":"2024-03-04T15:28:01.475240+0000","last_unstale":"2024-03-04T15:28:27.715166+0000","last_undegraded":"2024-03-04T15:28:27.715166+0000","last_fullsized":"2024-03-04T15:28:27.715166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:15:25.250136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717163+0000","last_change":"2024-03-04T15:27:59.467719+0000","last_active":"2024-03-04T15:28:27.717163+0000","last_peered":"2024-03-04T15:28:27.717163+0000","last_clean":"2024-03-04T15:28:27.717163+0000","last_became_active":"2024-03-04T15:27:59.467050+0000","last_became_peered":"2024-03-04T15:27:59.467050+0000","last_unstale":"2024-03-04T15:28:27.717163+0000","last_undegraded":"2024-03-04T15:28:27.717163+0000","last_fullsized":"2024-03-04T15:28:27.717163+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:51:20.688294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:29.832660+0000","last_change":"2024-03-04T15:27:47.908050+0000","last_active":"2024-03-04T15:28:29.832660+0000","last_peered":"2024-03-04T15:28:29.832660+0000","last_clean":"2024-03-04T15:28:29.832660+0000","last_became_active":"2024-03-04T15:27:47.904891+0000","last_became_peered":"2024-03-04T15:27:47.904891+0000","last_unstale":"2024-03-04T15:28:29.832660+0000","last_undegraded":"2024-03-04T15:28:29.832660+0000","last_fullsized":"2024-03-04T15:28:29.832660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:46.888899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:46.888899+0000","last_clean_scrub_stamp":"2024-03-04T15:27:46.888899+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:26:17.050613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718379+0000","last_change":"2024-03-04T15:28:03.483931+0000","last_active":"2024-03-04T15:28:27.718379+0000","last_peered":"2024-03-04T15:28:27.718379+0000","last_clean":"2024-03-04T15:28:27.718379+0000","last_became_active":"2024-03-04T15:28:03.482483+0000","last_became_peered":"2024-03-04T15:28:03.482483+0000","last_unstale":"2024-03-04T15:28:27.718379+0000","last_undegraded":"2024-03-04T15:28:27.718379+0000","last_fullsized":"2024-03-04T15:28:27.718379+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:51:35.527999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717090+0000","last_change":"2024-03-04T15:28:05.503166+0000","last_active":"2024-03-04T15:28:27.717090+0000","last_peered":"2024-03-04T15:28:27.717090+0000","last_clean":"2024-03-04T15:28:27.717090+0000","last_became_active":"2024-03-04T15:28:05.502068+0000","last_became_peered":"2024-03-04T15:28:05.502068+0000","last_unstale":"2024-03-04T15:28:27.717090+0000","last_undegraded":"2024-03-04T15:28:27.717090+0000","last_fullsized":"2024-03-04T15:28:27.717090+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:51:38.165836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714353+0000","last_change":"2024-03-04T15:28:01.478497+0000","last_active":"2024-03-04T15:28:27.714353+0000","last_peered":"2024-03-04T15:28:27.714353+0000","last_clean":"2024-03-04T15:28:27.714353+0000","last_became_active":"2024-03-04T15:28:01.478052+0000","last_became_peered":"2024-03-04T15:28:01.478052+0000","last_unstale":"2024-03-04T15:28:27.714353+0000","last_undegraded":"2024-03-04T15:28:27.714353+0000","last_fullsized":"2024-03-04T15:28:27.714353+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:41:18.463499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715501+0000","last_change":"2024-03-04T15:27:59.468589+0000","last_active":"2024-03-04T15:28:27.715501+0000","last_peered":"2024-03-04T15:28:27.715501+0000","last_clean":"2024-03-04T15:28:27.715501+0000","last_became_active":"2024-03-04T15:27:59.468254+0000","last_became_peered":"2024-03-04T15:27:59.468254+0000","last_unstale":"2024-03-04T15:28:27.715501+0000","last_undegraded":"2024-03-04T15:28:27.715501+0000","last_fullsized":"2024-03-04T15:28:27.715501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:28:56.422093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718731+0000","last_change":"2024-03-04T15:28:03.484421+0000","last_active":"2024-03-04T15:28:27.718731+0000","last_peered":"2024-03-04T15:28:27.718731+0000","last_clean":"2024-03-04T15:28:27.718731+0000","last_became_active":"2024-03-04T15:28:03.483959+0000","last_became_peered":"2024-03-04T15:28:03.483959+0000","last_unstale":"2024-03-04T15:28:27.718731+0000","last_undegraded":"2024-03-04T15:28:27.718731+0000","last_fullsized":"2024-03-04T15:28:27.718731+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:17:47.138144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716180+0000","last_change":"2024-03-04T15:28:05.501152+0000","last_active":"2024-03-04T15:28:27.716180+0000","last_peered":"2024-03-04T15:28:27.716180+0000","last_clean":"2024-03-04T15:28:27.716180+0000","last_became_active":"2024-03-04T15:28:05.500508+0000","last_became_peered":"2024-03-04T15:28:05.500508+0000","last_unstale":"2024-03-04T15:28:27.716180+0000","last_undegraded":"2024-03-04T15:28:27.716180+0000","last_fullsized":"2024-03-04T15:28:27.716180+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:03:48.967046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.721120+0000","last_change":"2024-03-04T15:27:59.467864+0000","last_active":"2024-03-04T15:28:27.721120+0000","last_peered":"2024-03-04T15:28:27.721120+0000","last_clean":"2024-03-04T15:28:27.721120+0000","last_became_active":"2024-03-04T15:27:59.467177+0000","last_became_peered":"2024-03-04T15:27:59.467177+0000","last_unstale":"2024-03-04T15:28:27.721120+0000","last_undegraded":"2024-03-04T15:28:27.721120+0000","last_fullsized":"2024-03-04T15:28:27.721120+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:02:07.360073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":80,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513534+0000","last_change":"2024-03-04T15:28:01.477896+0000","last_active":"2024-03-04T15:28:34.513534+0000","last_peered":"2024-03-04T15:28:34.513534+0000","last_clean":"2024-03-04T15:28:34.513534+0000","last_became_active":"2024-03-04T15:28:01.477412+0000","last_became_peered":"2024-03-04T15:28:01.477412+0000","last_unstale":"2024-03-04T15:28:34.513534+0000","last_undegraded":"2024-03-04T15:28:34.513534+0000","last_fullsized":"2024-03-04T15:28:34.513534+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:45:15.754629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513846+0000","last_change":"2024-03-04T15:28:03.484105+0000","last_active":"2024-03-04T15:28:34.513846+0000","last_peered":"2024-03-04T15:28:34.513846+0000","last_clean":"2024-03-04T15:28:34.513846+0000","last_became_active":"2024-03-04T15:28:03.483337+0000","last_became_peered":"2024-03-04T15:28:03.483337+0000","last_unstale":"2024-03-04T15:28:34.513846+0000","last_undegraded":"2024-03-04T15:28:34.513846+0000","last_fullsized":"2024-03-04T15:28:34.513846+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:02:09.706001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715644+0000","last_change":"2024-03-04T15:28:05.501304+0000","last_active":"2024-03-04T15:28:27.715644+0000","last_peered":"2024-03-04T15:28:27.715644+0000","last_clean":"2024-03-04T15:28:27.715644+0000","last_became_active":"2024-03-04T15:28:05.500387+0000","last_became_peered":"2024-03-04T15:28:05.500387+0000","last_unstale":"2024-03-04T15:28:27.715644+0000","last_undegraded":"2024-03-04T15:28:27.715644+0000","last_fullsized":"2024-03-04T15:28:27.715644+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:56:15.006474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'2","reported_seq":34,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717371+0000","last_change":"2024-03-04T15:27:59.465923+0000","last_active":"2024-03-04T15:28:27.717371+0000","last_peered":"2024-03-04T15:28:27.717371+0000","last_clean":"2024-03-04T15:28:27.717371+0000","last_became_active":"2024-03-04T15:27:59.464933+0000","last_became_peered":"2024-03-04T15:27:59.464933+0000","last_unstale":"2024-03-04T15:28:27.717371+0000","last_undegraded":"2024-03-04T15:28:27.717371+0000","last_fullsized":"2024-03-04T15:28:27.717371+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:34:45.241543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":921,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718232+0000","last_change":"2024-03-04T15:28:01.482357+0000","last_active":"2024-03-04T15:28:27.718232+0000","last_peered":"2024-03-04T15:28:27.718232+0000","last_clean":"2024-03-04T15:28:27.718232+0000","last_became_active":"2024-03-04T15:28:01.482041+0000","last_became_peered":"2024-03-04T15:28:01.482041+0000","last_unstale":"2024-03-04T15:28:27.718232+0000","last_undegraded":"2024-03-04T15:28:27.718232+0000","last_fullsized":"2024-03-04T15:28:27.718232+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:14:30.669292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513542+0000","last_change":"2024-03-04T15:28:03.487302+0000","last_active":"2024-03-04T15:28:34.513542+0000","last_peered":"2024-03-04T15:28:34.513542+0000","last_clean":"2024-03-04T15:28:34.513542+0000","last_became_active":"2024-03-04T15:28:03.486924+0000","last_became_peered":"2024-03-04T15:28:03.486924+0000","last_unstale":"2024-03-04T15:28:34.513542+0000","last_undegraded":"2024-03-04T15:28:34.513542+0000","last_fullsized":"2024-03-04T15:28:34.513542+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:08:27.367873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719068+0000","last_change":"2024-03-04T15:28:05.500430+0000","last_active":"2024-03-04T15:28:27.719068+0000","last_peered":"2024-03-04T15:28:27.719068+0000","last_clean":"2024-03-04T15:28:27.719068+0000","last_became_active":"2024-03-04T15:28:05.499895+0000","last_became_peered":"2024-03-04T15:28:05.499895+0000","last_unstale":"2024-03-04T15:28:27.719068+0000","last_undegraded":"2024-03-04T15:28:27.719068+0000","last_fullsized":"2024-03-04T15:28:27.719068+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:06:07.421134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718317+0000","last_change":"2024-03-04T15:27:59.468643+0000","last_active":"2024-03-04T15:28:27.718317+0000","last_peered":"2024-03-04T15:28:27.718317+0000","last_clean":"2024-03-04T15:28:27.718317+0000","last_became_active":"2024-03-04T15:27:59.468295+0000","last_became_peered":"2024-03-04T15:27:59.468295+0000","last_unstale":"2024-03-04T15:28:27.718317+0000","last_undegraded":"2024-03-04T15:28:27.718317+0000","last_fullsized":"2024-03-04T15:28:27.718317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:04:59.633181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717040+0000","last_change":"2024-03-04T15:28:01.481755+0000","last_active":"2024-03-04T15:28:27.717040+0000","last_peered":"2024-03-04T15:28:27.717040+0000","last_clean":"2024-03-04T15:28:27.717040+0000","last_became_active":"2024-03-04T15:28:01.481415+0000","last_became_peered":"2024-03-04T15:28:01.481415+0000","last_unstale":"2024-03-04T15:28:27.717040+0000","last_undegraded":"2024-03-04T15:28:27.717040+0000","last_fullsized":"2024-03-04T15:28:27.717040+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:15:22.852308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717916+0000","last_change":"2024-03-04T15:28:03.482727+0000","last_active":"2024-03-04T15:28:27.717916+0000","last_peered":"2024-03-04T15:28:27.717916+0000","last_clean":"2024-03-04T15:28:27.717916+0000","last_became_active":"2024-03-04T15:28:03.482446+0000","last_became_peered":"2024-03-04T15:28:03.482446+0000","last_unstale":"2024-03-04T15:28:27.717916+0000","last_undegraded":"2024-03-04T15:28:27.717916+0000","last_fullsized":"2024-03-04T15:28:27.717916+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:03:23.013026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716930+0000","last_change":"2024-03-04T15:28:05.502563+0000","last_active":"2024-03-04T15:28:27.716930+0000","last_peered":"2024-03-04T15:28:27.716930+0000","last_clean":"2024-03-04T15:28:27.716930+0000","last_became_active":"2024-03-04T15:28:05.501280+0000","last_became_peered":"2024-03-04T15:28:05.501280+0000","last_unstale":"2024-03-04T15:28:27.716930+0000","last_undegraded":"2024-03-04T15:28:27.716930+0000","last_fullsized":"2024-03-04T15:28:27.716930+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:18:52.239497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":36,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513756+0000","last_change":"2024-03-04T15:28:03.484438+0000","last_active":"2024-03-04T15:28:34.513756+0000","last_peered":"2024-03-04T15:28:34.513756+0000","last_clean":"2024-03-04T15:28:34.513756+0000","last_became_active":"2024-03-04T15:28:03.483987+0000","last_became_peered":"2024-03-04T15:28:03.483987+0000","last_unstale":"2024-03-04T15:28:34.513756+0000","last_undegraded":"2024-03-04T15:28:34.513756+0000","last_fullsized":"2024-03-04T15:28:34.513756+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:08:10.496103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715739+0000","last_change":"2024-03-04T15:27:59.467686+0000","last_active":"2024-03-04T15:28:27.715739+0000","last_peered":"2024-03-04T15:28:27.715739+0000","last_clean":"2024-03-04T15:28:27.715739+0000","last_became_active":"2024-03-04T15:27:59.467101+0000","last_became_peered":"2024-03-04T15:27:59.467101+0000","last_unstale":"2024-03-04T15:28:27.715739+0000","last_undegraded":"2024-03-04T15:28:27.715739+0000","last_fullsized":"2024-03-04T15:28:27.715739+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:43:33.088437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718608+0000","last_change":"2024-03-04T15:28:01.478029+0000","last_active":"2024-03-04T15:28:27.718608+0000","last_peered":"2024-03-04T15:28:27.718608+0000","last_clean":"2024-03-04T15:28:27.718608+0000","last_became_active":"2024-03-04T15:28:01.477687+0000","last_became_peered":"2024-03-04T15:28:01.477687+0000","last_unstale":"2024-03-04T15:28:27.718608+0000","last_undegraded":"2024-03-04T15:28:27.718608+0000","last_fullsized":"2024-03-04T15:28:27.718608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:39:21.992595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718943+0000","last_change":"2024-03-04T15:28:05.500075+0000","last_active":"2024-03-04T15:28:27.718943+0000","last_peered":"2024-03-04T15:28:27.718943+0000","last_clean":"2024-03-04T15:28:27.718943+0000","last_became_active":"2024-03-04T15:28:05.499647+0000","last_became_peered":"2024-03-04T15:28:05.499647+0000","last_unstale":"2024-03-04T15:28:27.718943+0000","last_undegraded":"2024-03-04T15:28:27.718943+0000","last_fullsized":"2024-03-04T15:28:27.718943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:51:20.536863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717946+0000","last_change":"2024-03-04T15:28:03.488396+0000","last_active":"2024-03-04T15:28:27.717946+0000","last_peered":"2024-03-04T15:28:27.717946+0000","last_clean":"2024-03-04T15:28:27.717946+0000","last_became_active":"2024-03-04T15:28:03.488095+0000","last_became_peered":"2024-03-04T15:28:03.488095+0000","last_unstale":"2024-03-04T15:28:27.717946+0000","last_undegraded":"2024-03-04T15:28:27.717946+0000","last_fullsized":"2024-03-04T15:28:27.717946+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T17:25:18.427286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714087+0000","last_change":"2024-03-04T15:27:59.466478+0000","last_active":"2024-03-04T15:28:27.714087+0000","last_peered":"2024-03-04T15:28:27.714087+0000","last_clean":"2024-03-04T15:28:27.714087+0000","last_became_active":"2024-03-04T15:27:59.465601+0000","last_became_peered":"2024-03-04T15:27:59.465601+0000","last_unstale":"2024-03-04T15:28:27.714087+0000","last_undegraded":"2024-03-04T15:28:27.714087+0000","last_fullsized":"2024-03-04T15:28:27.714087+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:21:11.816176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718178+0000","last_change":"2024-03-04T15:28:01.477492+0000","last_active":"2024-03-04T15:28:27.718178+0000","last_peered":"2024-03-04T15:28:27.718178+0000","last_clean":"2024-03-04T15:28:27.718178+0000","last_became_active":"2024-03-04T15:28:01.476771+0000","last_became_peered":"2024-03-04T15:28:01.476771+0000","last_unstale":"2024-03-04T15:28:27.718178+0000","last_undegraded":"2024-03-04T15:28:27.718178+0000","last_fullsized":"2024-03-04T15:28:27.718178+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:35:49.229666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720844+0000","last_change":"2024-03-04T15:28:05.500469+0000","last_active":"2024-03-04T15:28:27.720844+0000","last_peered":"2024-03-04T15:28:27.720844+0000","last_clean":"2024-03-04T15:28:27.720844+0000","last_became_active":"2024-03-04T15:28:05.499905+0000","last_became_peered":"2024-03-04T15:28:05.499905+0000","last_unstale":"2024-03-04T15:28:27.720844+0000","last_undegraded":"2024-03-04T15:28:27.720844+0000","last_fullsized":"2024-03-04T15:28:27.720844+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T15:34:54.428688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719697+0000","last_change":"2024-03-04T15:27:59.464685+0000","last_active":"2024-03-04T15:28:27.719697+0000","last_peered":"2024-03-04T15:28:27.719697+0000","last_clean":"2024-03-04T15:28:27.719697+0000","last_became_active":"2024-03-04T15:27:59.464296+0000","last_became_peered":"2024-03-04T15:27:59.464296+0000","last_unstale":"2024-03-04T15:28:27.719697+0000","last_undegraded":"2024-03-04T15:28:27.719697+0000","last_fullsized":"2024-03-04T15:28:27.719697+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:09:43.453889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714924+0000","last_change":"2024-03-04T15:28:01.478023+0000","last_active":"2024-03-04T15:28:27.714924+0000","last_peered":"2024-03-04T15:28:27.714924+0000","last_clean":"2024-03-04T15:28:27.714924+0000","last_became_active":"2024-03-04T15:28:01.477361+0000","last_became_peered":"2024-03-04T15:28:01.477361+0000","last_unstale":"2024-03-04T15:28:27.714924+0000","last_undegraded":"2024-03-04T15:28:27.714924+0000","last_fullsized":"2024-03-04T15:28:27.714924+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:18:04.146768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717223+0000","last_change":"2024-03-04T15:28:03.486367+0000","last_active":"2024-03-04T15:28:27.717223+0000","last_peered":"2024-03-04T15:28:27.717223+0000","last_clean":"2024-03-04T15:28:27.717223+0000","last_became_active":"2024-03-04T15:28:03.486055+0000","last_became_peered":"2024-03-04T15:28:03.486055+0000","last_unstale":"2024-03-04T15:28:27.717223+0000","last_undegraded":"2024-03-04T15:28:27.717223+0000","last_fullsized":"2024-03-04T15:28:27.717223+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:39:19.640433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716689+0000","last_change":"2024-03-04T15:28:05.499181+0000","last_active":"2024-03-04T15:28:27.716689+0000","last_peered":"2024-03-04T15:28:27.716689+0000","last_clean":"2024-03-04T15:28:27.716689+0000","last_became_active":"2024-03-04T15:28:05.498315+0000","last_became_peered":"2024-03-04T15:28:05.498315+0000","last_unstale":"2024-03-04T15:28:27.716689+0000","last_undegraded":"2024-03-04T15:28:27.716689+0000","last_fullsized":"2024-03-04T15:28:27.716689+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T16:21:08.070756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:34.513811+0000","last_change":"2024-03-04T15:28:03.484301+0000","last_active":"2024-03-04T15:28:34.513811+0000","last_peered":"2024-03-04T15:28:34.513811+0000","last_clean":"2024-03-04T15:28:34.513811+0000","last_became_active":"2024-03-04T15:28:03.483239+0000","last_became_peered":"2024-03-04T15:28:03.483239+0000","last_unstale":"2024-03-04T15:28:34.513811+0000","last_undegraded":"2024-03-04T15:28:34.513811+0000","last_fullsized":"2024-03-04T15:28:34.513811+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:57:55.704394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718744+0000","last_change":"2024-03-04T15:27:59.465486+0000","last_active":"2024-03-04T15:28:27.718744+0000","last_peered":"2024-03-04T15:28:27.718744+0000","last_clean":"2024-03-04T15:28:27.718744+0000","last_became_active":"2024-03-04T15:27:59.465171+0000","last_became_peered":"2024-03-04T15:27:59.465171+0000","last_unstale":"2024-03-04T15:28:27.718744+0000","last_undegraded":"2024-03-04T15:28:27.718744+0000","last_fullsized":"2024-03-04T15:28:27.718744+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T22:56:41.292712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716536+0000","last_change":"2024-03-04T15:28:01.477758+0000","last_active":"2024-03-04T15:28:27.716536+0000","last_peered":"2024-03-04T15:28:27.716536+0000","last_clean":"2024-03-04T15:28:27.716536+0000","last_became_active":"2024-03-04T15:28:01.477284+0000","last_became_peered":"2024-03-04T15:28:01.477284+0000","last_unstale":"2024-03-04T15:28:27.716536+0000","last_undegraded":"2024-03-04T15:28:27.716536+0000","last_fullsized":"2024-03-04T15:28:27.716536+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T21:37:32.810310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718099+0000","last_change":"2024-03-04T15:28:05.500368+0000","last_active":"2024-03-04T15:28:27.718099+0000","last_peered":"2024-03-04T15:28:27.718099+0000","last_clean":"2024-03-04T15:28:27.718099+0000","last_became_active":"2024-03-04T15:28:05.499835+0000","last_became_peered":"2024-03-04T15:28:05.499835+0000","last_unstale":"2024-03-04T15:28:27.718099+0000","last_undegraded":"2024-03-04T15:28:27.718099+0000","last_fullsized":"2024-03-04T15:28:27.718099+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:40:34.208403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.719395+0000","last_change":"2024-03-04T15:27:59.465303+0000","last_active":"2024-03-04T15:28:27.719395+0000","last_peered":"2024-03-04T15:28:27.719395+0000","last_clean":"2024-03-04T15:28:27.719395+0000","last_became_active":"2024-03-04T15:27:59.464756+0000","last_became_peered":"2024-03-04T15:27:59.464756+0000","last_unstale":"2024-03-04T15:28:27.719395+0000","last_undegraded":"2024-03-04T15:28:27.719395+0000","last_fullsized":"2024-03-04T15:28:27.719395+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T02:25:13.372166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718617+0000","last_change":"2024-03-04T15:28:01.477777+0000","last_active":"2024-03-04T15:28:27.718617+0000","last_peered":"2024-03-04T15:28:27.718617+0000","last_clean":"2024-03-04T15:28:27.718617+0000","last_became_active":"2024-03-04T15:28:01.477385+0000","last_became_peered":"2024-03-04T15:28:01.477385+0000","last_unstale":"2024-03-04T15:28:27.718617+0000","last_undegraded":"2024-03-04T15:28:27.718617+0000","last_fullsized":"2024-03-04T15:28:27.718617+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:53:33.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716056+0000","last_change":"2024-03-04T15:28:03.484155+0000","last_active":"2024-03-04T15:28:27.716056+0000","last_peered":"2024-03-04T15:28:27.716056+0000","last_clean":"2024-03-04T15:28:27.716056+0000","last_became_active":"2024-03-04T15:28:03.483252+0000","last_became_peered":"2024-03-04T15:28:03.483252+0000","last_unstale":"2024-03-04T15:28:27.716056+0000","last_undegraded":"2024-03-04T15:28:27.716056+0000","last_fullsized":"2024-03-04T15:28:27.716056+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T01:00:16.640769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.713483+0000","last_change":"2024-03-04T15:28:05.497924+0000","last_active":"2024-03-04T15:28:27.713483+0000","last_peered":"2024-03-04T15:28:27.713483+0000","last_clean":"2024-03-04T15:28:27.713483+0000","last_became_active":"2024-03-04T15:28:05.497309+0000","last_became_peered":"2024-03-04T15:28:05.497309+0000","last_unstale":"2024-03-04T15:28:27.713483+0000","last_undegraded":"2024-03-04T15:28:27.713483+0000","last_fullsized":"2024-03-04T15:28:27.713483+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:33:45.755167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.720986+0000","last_change":"2024-03-04T15:27:59.468001+0000","last_active":"2024-03-04T15:28:27.720986+0000","last_peered":"2024-03-04T15:28:27.720986+0000","last_clean":"2024-03-04T15:28:27.720986+0000","last_became_active":"2024-03-04T15:27:59.467484+0000","last_became_peered":"2024-03-04T15:27:59.467484+0000","last_unstale":"2024-03-04T15:28:27.720986+0000","last_undegraded":"2024-03-04T15:28:27.720986+0000","last_fullsized":"2024-03-04T15:28:27.720986+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:14:20.983825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.717169+0000","last_change":"2024-03-04T15:28:01.478952+0000","last_active":"2024-03-04T15:28:27.717169+0000","last_peered":"2024-03-04T15:28:27.717169+0000","last_clean":"2024-03-04T15:28:27.717169+0000","last_became_active":"2024-03-04T15:28:01.477787+0000","last_became_peered":"2024-03-04T15:28:01.477787+0000","last_unstale":"2024-03-04T15:28:27.717169+0000","last_undegraded":"2024-03-04T15:28:27.717169+0000","last_fullsized":"2024-03-04T15:28:27.717169+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T23:45:51.535725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.714362+0000","last_change":"2024-03-04T15:28:03.483230+0000","last_active":"2024-03-04T15:28:27.714362+0000","last_peered":"2024-03-04T15:28:27.714362+0000","last_clean":"2024-03-04T15:28:27.714362+0000","last_became_active":"2024-03-04T15:28:03.482687+0000","last_became_peered":"2024-03-04T15:28:03.482687+0000","last_unstale":"2024-03-04T15:28:27.714362+0000","last_undegraded":"2024-03-04T15:28:27.714362+0000","last_fullsized":"2024-03-04T15:28:27.714362+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:45:01.070235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.715490+0000","last_change":"2024-03-04T15:28:05.502635+0000","last_active":"2024-03-04T15:28:27.715490+0000","last_peered":"2024-03-04T15:28:27.715490+0000","last_clean":"2024-03-04T15:28:27.715490+0000","last_became_active":"2024-03-04T15:28:05.502164+0000","last_became_peered":"2024-03-04T15:28:05.502164+0000","last_unstale":"2024-03-04T15:28:27.715490+0000","last_undegraded":"2024-03-04T15:28:27.715490+0000","last_fullsized":"2024-03-04T15:28:27.715490+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:04.477723+0000","last_clean_scrub_stamp":"2024-03-04T15:28:04.477723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T19:24:05.654891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.716950+0000","last_change":"2024-03-04T15:27:59.465157+0000","last_active":"2024-03-04T15:28:27.716950+0000","last_peered":"2024-03-04T15:28:27.716950+0000","last_clean":"2024-03-04T15:28:27.716950+0000","last_became_active":"2024-03-04T15:27:59.464305+0000","last_became_peered":"2024-03-04T15:27:59.464305+0000","last_unstale":"2024-03-04T15:28:27.716950+0000","last_undegraded":"2024-03-04T15:28:27.716950+0000","last_fullsized":"2024-03-04T15:28:27.716950+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:27:58.443941+0000","last_clean_scrub_stamp":"2024-03-04T15:27:58.443941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T20:21:29.496585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718379+0000","last_change":"2024-03-04T15:28:01.479269+0000","last_active":"2024-03-04T15:28:27.718379+0000","last_peered":"2024-03-04T15:28:27.718379+0000","last_clean":"2024-03-04T15:28:27.718379+0000","last_became_active":"2024-03-04T15:28:01.478784+0000","last_became_peered":"2024-03-04T15:28:01.478784+0000","last_unstale":"2024-03-04T15:28:27.718379+0000","last_undegraded":"2024-03-04T15:28:27.718379+0000","last_fullsized":"2024-03-04T15:28:27.718379+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:00.456414+0000","last_clean_scrub_stamp":"2024-03-04T15:28:00.456414+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-05T18:45:21.598972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-04T15:28:27.718430+0000","last_change":"2024-03-04T15:28:03.483707+0000","last_active":"2024-03-04T15:28:27.718430+0000","last_peered":"2024-03-04T15:28:27.718430+0000","last_clean":"2024-03-04T15:28:27.718430+0000","last_became_active":"2024-03-04T15:28:03.483002+0000","last_became_peered":"2024-03-04T15:28:03.483002+0000","last_unstale":"2024-03-04T15:28:27.718430+0000","last_undegraded":"2024-03-04T15:28:27.718430+0000","last_fullsized":"2024-03-04T15:28:27.718430+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T15:28:02.467055+0000","last_clean_scrub_stamp":"2024-03-04T15:28:02.467055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-06T00:19:32.170341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704688,"statfs":{"total":95990841344,"available":95953600512,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T15:28:45.128 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T15:28:45.129 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T15:28:45.129 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-04T15:28:45.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-04T15:28:45.723 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:45 smithi053 ceph-mon[25738]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:45.723 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:45 smithi053 ceph-mon[25738]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:28:45.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:45 smithi064 ceph-mon[30297]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:45.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:45 smithi064 ceph-mon[30297]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:28:45.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:45 smithi160 ceph-mon[30717]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:45.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:45 smithi160 ceph-mon[30717]: from='client.14532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:28:47.356 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:47.732 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:47 smithi053 ceph-mon[25738]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:47.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:47 smithi064 ceph-mon[30297]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:47.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:47 smithi160 ceph-mon[30717]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:48.603 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 podman[54524]: 2024-03-04 15:28:48.393441868 +0000 UTC m=+0.223114041 container create 96b844979cbdcf4d4e61e1e5201548242e0b4e268106f111e1d4ff28bafecfc1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:28:49.086 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:48] ENGINE Bus STOPPING 2024-03-04T15:28:49.086 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T15:28:49.086 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:48] ENGINE Bus STOPPED 2024-03-04T15:28:49.086 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:48] ENGINE Bus STARTING 2024-03-04T15:28:49.086 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:49] ENGINE Serving on http://:::9283 2024-03-04T15:28:49.086 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:28:49] ENGINE Bus STARTED 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 podman[54524]: 2024-03-04 15:28:48.737909102 +0000 UTC m=+0.567581233 container init 96b844979cbdcf4d4e61e1e5201548242e0b4e268106f111e1d4ff28bafecfc1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 podman[54524]: 2024-03-04 15:28:48.74348071 +0000 UTC m=+0.573152841 container start 96b844979cbdcf4d4e61e1e5201548242e0b4e268106f111e1d4ff28bafecfc1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.756Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.756Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.757Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.53 port=9094 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.758Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.781Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.781Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.784Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:48.784Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 bash[54524]: 96b844979cbdcf4d4e61e1e5201548242e0b4e268106f111e1d4ff28bafecfc1 2024-03-04T15:28:49.087 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:48 smithi053 systemd[1]: Started Ceph alertmanager.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:28:49.719 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:49.719 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.719 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.719 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.720 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.720 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.720 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.720 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:28:49.720 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:49 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:28:49.773 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:49 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:28:49.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:49 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:49.999 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-04T15:28:49.999 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T15:28:50.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:50 smithi053 ceph-mon[25738]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T15:28:50.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:50 smithi053 ceph-mon[25738]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:28:50.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:50 smithi053 ceph-mon[25738]: Deploying daemon grafana.a on smithi064 2024-03-04T15:28:50.558 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:50 smithi053 ceph-mon[25738]: from='client.? 172.21.15.53:0/3116041956' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T15:28:50.559 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-04T15:28:50.560 INFO:tasks.cephadm:Setup complete, yielding 2024-03-04T15:28:50.560 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-04T15:28:50.571 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi053.front.sepia.ceph.com 2024-03-04T15:28:50.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-04T15:28:50.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:50 smithi064 ceph-mon[30297]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T15:28:50.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:50 smithi064 ceph-mon[30297]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:28:50.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:50 smithi064 ceph-mon[30297]: Deploying daemon grafana.a on smithi064 2024-03-04T15:28:50.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:50 smithi064 ceph-mon[30297]: from='client.? 172.21.15.53:0/3116041956' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T15:28:50.815 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:50.886 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:50 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:50.758Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000520859s 2024-03-04T15:28:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:50 smithi160 ceph-mon[30717]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T15:28:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:50 smithi160 ceph-mon[30717]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T15:28:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:50 smithi160 ceph-mon[30717]: Deploying daemon grafana.a on smithi064 2024-03-04T15:28:50.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:50 smithi160 ceph-mon[30717]: from='client.? 172.21.15.53:0/3116041956' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T15:28:51.512 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:51 smithi053 ceph-mon[25738]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:51.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:51 smithi064 ceph-mon[30297]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:51.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:51 smithi160 ceph-mon[30717]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:52.257 INFO:teuthology.orchestra.run.smithi053.stdout:167 167 2024-03-04T15:28:52.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:28:52 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ::ffff:172.21.15.64 - - [04/Mar/2024:15:28:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:28:52.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch status' 2024-03-04T15:28:53.208 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:53.564 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:53 smithi053 ceph-mon[25738]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:53.564 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:53 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:53.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:53 smithi064 ceph-mon[30297]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:53.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:53 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:53.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:53 smithi160 ceph-mon[30717]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:53.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:53 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:28:55.003 INFO:teuthology.orchestra.run.smithi053.stdout:Backend: cephadm 2024-03-04T15:28:55.003 INFO:teuthology.orchestra.run.smithi053.stdout:Available: Yes 2024-03-04T15:28:55.003 INFO:teuthology.orchestra.run.smithi053.stdout:Paused: No 2024-03-04T15:28:55.566 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:55 smithi053 ceph-mon[25738]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:55.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-04T15:28:55.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:55 smithi064 ceph-mon[30297]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:55.837 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:55.964 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:55 smithi160 ceph-mon[30717]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:56.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:56 smithi053 ceph-mon[25738]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:56.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:56 smithi064 ceph-mon[30297]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:56.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:56 smithi160 ceph-mon[30717]: from='client.14544 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:57.586 INFO:teuthology.orchestra.run.smithi053.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-04T15:28:57.586 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager.a smithi053 *:9093,9094 starting - - - - 2024-03-04T15:28:57.586 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.x smithi064 *:8443,8765 running (2m) 27s ago 2m 448M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4fbf03a2016d 2024-03-04T15:28:57.586 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.y smithi053 *:9283,8765 running (4m) 22s ago 4m 480M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 5724c6d25cb1 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:mgr.z smithi160 *:8443,8765 running (2m) 28s ago 2m 447M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 89ece9067207 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:mon.a smithi053 running (4m) 22s ago 4m 50.9M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4625c153d198 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:mon.b smithi064 running (2m) 27s ago 2m 42.1M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 4526efd1013b 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:mon.c smithi160 running (2m) 28s ago 2m 43.2M 2048M 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 19832f78866f 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:osd.0 smithi053 running (2m) 22s ago 2m 62.0M 15.6G 19.0.0-1826-g9ca0b5de 0d1497d8ab5e bd641edd1c6d 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:osd.1 smithi064 running (102s) 27s ago 102s 50.5M 15.6G 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 372d4b963d5e 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:osd.2 smithi160 running (77s) 28s ago 77s 50.5M 16.6G 19.0.0-1826-g9ca0b5de 0d1497d8ab5e d525123c8d9b 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus.a smithi064 *:9095 running (39s) 27s ago 38s 25.4M - 2.43.0 a07b618ecd1d 71e0e568b872 2024-03-04T15:28:57.587 INFO:teuthology.orchestra.run.smithi053.stdout:rgw.foo.a smithi053 *:80 running (60s) 22s ago 59s 76.2M - 19.0.0-1826-g9ca0b5de 0d1497d8ab5e 0062d4ed96bd 2024-03-04T15:28:57.599 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:57 smithi053 ceph-mon[25738]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:57.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:57 smithi064 ceph-mon[30297]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:57.968 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:57 smithi160 ceph-mon[30717]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:58.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-04T15:28:58.396 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:28:58.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:58 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:58.855 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:58 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:58.856 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:28:58 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:28:58.760Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002692522s 2024-03-04T15:28:58.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:58 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T15:28:59.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:59 smithi053 ceph-mon[25738]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:59.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:28:59 smithi053 ceph-mon[25738]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:59.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:59 smithi064 ceph-mon[30297]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:59.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:28:59 smithi064 ceph-mon[30297]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:28:59.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:59 smithi160 ceph-mon[30717]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:28:59.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:28:59 smithi160 ceph-mon[30717]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:alertmanager ?:9093,9094 0/1 - 57s smithi053=a;count:1 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:grafana ?:3000 0/1 - 54s smithi064=a;count:1 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:mgr 3/3 30s ago 2m smithi053=y;smithi064=x;smithi160=z;count:3 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:mon 3/3 30s ago 2m smithi053:172.21.15.53=a;smithi064:172.21.15.64=b;smithi160:172.21.15.160=c;count:3 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:node-exporter ?:9100 0/3 - 60s smithi053=a;smithi064=b;smithi160=c;count:3 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:osd 3 30s ago - 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:prometheus ?:9095 1/1 29s ago 63s smithi064=a;count:1 2024-03-04T15:29:00.159 INFO:teuthology.orchestra.run.smithi053.stdout:rgw.foo.a ?:80 1/1 24s ago 62s smithi053=foo.a;count:1 2024-03-04T15:29:00.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-04T15:29:00.995 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:29:01.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:01 smithi064 ceph-mon[30297]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:01.522 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:01 smithi064 ceph-mon[30297]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:01.780 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:01 smithi053 ceph-mon[25738]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:01.780 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:01 smithi053 ceph-mon[25738]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:01.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:01 smithi160 ceph-mon[30717]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:01.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:01 smithi160 ceph-mon[30717]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:02.635 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:02 smithi053 ceph-mon[25738]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:02.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:02 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ::ffff:172.21.15.64 - - [04/Mar/2024:15:29:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:29:02.725 INFO:teuthology.orchestra.run.smithi053.stdout:HOST ADDR LABELS STATUS 2024-03-04T15:29:02.725 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 172.21.15.53 2024-03-04T15:29:02.725 INFO:teuthology.orchestra.run.smithi053.stdout:smithi064 172.21.15.64 2024-03-04T15:29:02.725 INFO:teuthology.orchestra.run.smithi053.stdout:smithi160 172.21.15.160 2024-03-04T15:29:02.725 INFO:teuthology.orchestra.run.smithi053.stdout:3 hosts in cluster 2024-03-04T15:29:02.728 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:02 smithi064 ceph-mon[30297]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:02.965 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:02 smithi160 ceph-mon[30717]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:03.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-04T15:29:03.589 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:03 smithi064 ceph-mon[30297]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:03.669 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:29:03.790 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:03 smithi053 ceph-mon[25738]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:03.901 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:03 smithi160 ceph-mon[30717]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:04.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 podman[42821]: 2024-03-04 15:29:04.238999296 +0000 UTC m=+0.347290822 container start 05b263358208d82bebd4d3afcb741fd762b42ba9f818e7d4af0963b6391c6864 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base rhel8, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, vcs-type=git, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8) 2024-03-04T15:29:04.570 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 bash[42821]: 05b263358208d82bebd4d3afcb741fd762b42ba9f818e7d4af0963b6391c6864 2024-03-04T15:29:04.570 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 systemd[1]: Started Ceph grafana.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.395838344Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396039096Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396059914Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396068661Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396074974Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396080853Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.3960866Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396092444Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396098428Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.3961041Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396110601Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396116124Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-04T15:29:04.571 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.396121795Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-04T15:29:04.572 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=settings t=2024-03-04T15:29:04.39612746Z level=info msg="App mode production" 2024-03-04T15:29:04.572 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=sqlstore t=2024-03-04T15:29:04.396175119Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-04T15:29:04.572 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=sqlstore t=2024-03-04T15:29:04.39619308Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-04T15:29:04.572 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.397043859Z level=info msg="Starting DB migrations" 2024-03-04T15:29:04.572 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.397123413Z level=info msg="Executing migration" id="create migration_log table" 2024-03-04T15:29:04.573 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.397652415Z level=info msg="Executing migration" id="create user table" 2024-03-04T15:29:04.573 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.398078561Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-04T15:29:04.573 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.398558887Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-04T15:29:04.573 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.399033793Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-04T15:29:04.573 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.399633134Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-04T15:29:04.573 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.400016477Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-04T15:29:04.573 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.400735535Z level=info msg="Executing migration" id="create user table v2" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.401148957Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.4015762Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.402016093Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.402402457Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.402761573Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.403216121Z level=info msg="Executing migration" id="Update user table charset" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.403547557Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.403961954Z level=info msg="Executing migration" id="Add missing user data" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.404351397Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.404773004Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.405224257Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.405672821Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-04T15:29:04.574 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.406854781Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.407328547Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.407763354Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.408200772Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.409074069Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.40997546Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.410836023Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.411613846Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.412470551Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.413294433Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.414132469Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.414916848Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.415766366Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-04T15:29:04.575 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.416615083Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.417468836Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.418320773Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.419246158Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.420052396Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.420896748Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.421749673Z level=info msg="Executing migration" id="create star table" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.422178395Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.42310453Z level=info msg="Executing migration" id="create org table v1" 2024-03-04T15:29:04.576 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.423951721Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.424784801Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.425186371Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.426083492Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.426924907Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.42739795Z level=info msg="Executing migration" id="Update org table charset" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.428244673Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.429123772Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.430013053Z level=info msg="Executing migration" id="create dashboard table" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.432820651Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.435560855Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.438417347Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.441240672Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.44420999Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-04T15:29:04.577 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.446975953Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.449801509Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.452617827Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.455503452Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.458303812Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.461071443Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.46397948Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.466696813Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.469599274Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.472502086Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.475384715Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-04T15:29:04.578 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.478105419Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.480991155Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.483627323Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.486312731Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.488981337Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.489441694Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.492115195Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.494860188Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.497557199Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.500314964Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.503029429Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.505775384Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.50847278Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.511206338Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-04T15:29:04.579 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.513885953Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.516431535Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.51920967Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.52186326Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.524510025Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.527130228Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.52970147Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.532505109Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.533359829Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.534246025Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.535095426Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.535934632Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.536788886Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.537695314Z level=info msg="Executing migration" id="create data_source table" 2024-03-04T15:29:04.580 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.53853206Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.539386915Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.54023999Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.541055381Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.541911216Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.542921197Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.543697011Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.544543141Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.545395897Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.5461756Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.547038137Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.54791302Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.548791506Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.549649107Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.550496262Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-04T15:29:04.581 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.551343528Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.552207912Z level=info msg="Executing migration" id="Add uid column" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.553061731Z level=info msg="Executing migration" id="Update uid value" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.553911792Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.554791291Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.555646854Z level=info msg="Executing migration" id="create api_key table" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.55649865Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.5573506Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.558240715Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.559097358Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.559971816Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.560777289Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.561627136Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.56266542Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-04T15:29:04.582 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.565636455Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-04T15:29:04.583 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.568449444Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-04T15:29:04.583 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.571538553Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-04T15:29:04.583 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.572130782Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.572555966Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.575371927Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.575678743Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.578221702Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.580974302Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.581413855Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.584373785Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-04T15:29:04.823 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.584980017Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.587106164Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.589844043Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.590279125Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.592993282Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.595855949Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.598599148Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.601347326Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.604075831Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.606831584Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.609503511Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.60984271Z level=info msg="Executing migration" id="create quota table v1" 2024-03-04T15:29:04.824 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.6124153Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.615223666Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.617946415Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.618396048Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.621110912Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.623726018Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.626457815Z level=info msg="Executing migration" id="create session table" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.627218696Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.629916592Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.632528831Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.635103059Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.635520286Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.636369678Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-04T15:29:04.825 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.636695496Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-04T15:29:04.826 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.637213856Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-04T15:29:04.826 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.637558873Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-04T15:29:04.826 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.638024103Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-04T15:29:04.826 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.638338548Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-04T15:29:04.826 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.639165301Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-04T15:29:04.827 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.639506368Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-04T15:29:04.827 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.640159315Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-04T15:29:04.827 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.640954852Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-04T15:29:04.827 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.641294395Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.642157332Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.643064682Z level=info msg="Executing migration" id="create alert table v1" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.643825625Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.644347049Z level=info msg="Executing migration" id="add index alert state" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.645081183Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.645958101Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.646817878Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.647676374Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.648522192Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.649915398Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.650772969Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.651200154Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.652050459Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-04T15:29:04.828 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.652898047Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.653760049Z level=info msg="Executing migration" id="Add column is_default" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.654389292Z level=info msg="Executing migration" id="Add column frequency" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.655239773Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.656082471Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.656985879Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.657403099Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.658260838Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.65856228Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.659134474Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.660023869Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.660419663Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.663387034Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.663817805Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.668986157Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-04T15:29:04.829 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.671777711Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.672123058Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.674897941Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.677804474Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.680611675Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.680921999Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.683756763Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.68645045Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.689296212Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.68972683Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-04T15:29:04.830 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.692551535Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.695311337Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.69811693Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.698573909Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.70147012Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.704181239Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.706874018Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.707298948Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.710063896Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.712840001Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.715385718Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.718081659Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.720767868Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.723528357Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.726190743Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-04T15:29:04.831 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.72652281Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.729300074Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.729624025Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.731857584Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.734354416Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.734797011Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.73741707Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.738253826Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.73908036Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.739510316Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.740324127Z level=info msg="Executing migration" id="Move region to single row" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.740698027Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.741144064Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.741594167Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-04T15:29:04.832 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.742008699Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.742432656Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.7428118Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.743236679Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.743658743Z level=info msg="Executing migration" id="create test_data table" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.74447214Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.744888255Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.745314269Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.746144396Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.746998646Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.747772109Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.748042462Z level=info msg="Executing migration" id="create team table" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.748439624Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.74892688Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.749791525Z level=info msg="Executing migration" id="create team member table" 2024-03-04T15:29:04.833 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.750668677Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.751099227Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.75193921Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.752786032Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.753632596Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.754367317Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.755222035Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.75606542Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.756979463Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.757456891Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.757893717Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.758775419Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.759223473Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.759641638Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-04T15:29:04.834 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.76007335Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.760476893Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.760753852Z level=info msg="Executing migration" id="create tag table" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.761146802Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.761564053Z level=info msg="Executing migration" id="create login attempt table" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.761939479Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.76236583Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.762740814Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.764633506Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.765062416Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.765483899Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.765775695Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.76614231Z level=info msg="Executing migration" id="create user auth table" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.766520276Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-04T15:29:04.835 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.766937219Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.767239152Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.767950911Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.768689039Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.769428389Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.770188122Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.770611507Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.771350726Z level=info msg="Executing migration" id="create server_lock table" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.771739149Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.772175939Z level=info msg="Executing migration" id="create user auth token table" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.77260041Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.773392528Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.77382137Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.774274134Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-04T15:29:04.836 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.775024331Z level=info msg="Executing migration" id="create cache_data table" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.775430284Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.775850261Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.776272552Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.777106839Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.777407113Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.777682067Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.778120122Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.780918341Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.78369424Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.786525312Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.786980135Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.789976904Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.792770906Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.795587221Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-04T15:29:04.837 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.798404934Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.801165285Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.804035946Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.806781163Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.809632706Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.812540964Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.815529948Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.818267015Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.821216723Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-04T15:29:04.838 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.823825813Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-04T15:29:05.073 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.826555775Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-04T15:29:05.073 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.829443968Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.832133012Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.834831447Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.838265987Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.841634493Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.844277287Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.846821939Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.849480055Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.849921885Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.852773205Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.855535665Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.858132498Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.860940465Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-04T15:29:05.074 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.863563531Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.866220116Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.868823648Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.871645784Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.874253421Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.877071Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.877988861Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.878809343Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.879658392Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.882675673Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.885448841Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.88828266Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.890796755Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-04T15:29:05.075 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.893625218Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.896396612Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.899187202Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.902039078Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.904921066Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.907646663Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.910547856Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.913042417Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-04T15:29:05.076 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.915871911Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-04T15:29:05.077 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.918440785Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-04T15:29:05.077 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.921113156Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-04T15:29:05.077 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.92394294Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-04T15:29:05.077 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.926783177Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-04T15:29:05.077 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.92952408Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.932240938Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.934936996Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.937444821Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.940140723Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.9429083Z level=info msg="Executing migration" id="create alert_image table" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.9455513Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.948083063Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.95060321Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.953547443Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.954267951Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.954348458Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.955099407Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.956083738Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-04T15:29:05.078 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.956741836Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.956856907Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.956986072Z level=info msg="alerts found to migrate" alerts=0 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.957604359Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.958500107Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.959457295Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.96018291Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.961203999Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.961845354Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-04T15:29:05.079 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.962832492Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.963558735Z level=info msg="Executing migration" id="create data_keys table" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.964453608Z level=info msg="Executing migration" id="create secrets table" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.965319746Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.969665343Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.970579104Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.971423962Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.975702317Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.98002964Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.980994052Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.981905471Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.982590563Z level=info msg="Executing migration" id="create permission table" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.983514507Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-04T15:29:05.080 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.984473435Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.985215162Z level=info msg="Executing migration" id="create role table" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.986057929Z level=info msg="Executing migration" id="add column display_name" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.986990353Z level=info msg="Executing migration" id="add column group_name" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.987912115Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.988771434Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.989661541Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.990679835Z level=info msg="Executing migration" id="create team role table" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.991432532Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.992429576Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.993136992Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.994192777Z level=info msg="Executing migration" id="create user role table" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.997092692Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:04 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:04.999926068Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.002816212Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-04T15:29:05.081 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.005531044Z level=info msg="Executing migration" id="create builtin role table" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.008284105Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.01120911Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.014100502Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.017031443Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.019719598Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.022349754Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.025185517Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.028140615Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.03088088Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.033631186Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.036362297Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.039023527Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-04T15:29:05.082 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.041858853Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.044511694Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.047294348Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.050000057Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.052872093Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.055492895Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.058340733Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.060902692Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.063834171Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.066411812Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.069160221Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-04T15:29:05.083 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.071998049Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-04T15:29:05.381 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.074551972Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.077261827Z level=info msg="Executing migration" id="add correlation config column" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.080018936Z level=info msg="Executing migration" id="create entity_events table" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.082867797Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.085487583Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.085674897Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.088169341Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.088266544Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.09103194Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.091812537Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.092661207Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.093688674Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.094414245Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.095245962Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-04T15:29:05.382 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.096146018Z level=info msg="Executing migration" id="Drop public config table" 2024-03-04T15:29:05.383 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.097154259Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-04T15:29:05.383 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.09784551Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-04T15:29:05.383 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.098691002Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-04T15:29:05.383 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.099638488Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-04T15:29:05.383 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.100351063Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.103382842Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.104469078Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.105500381Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.106314938Z level=info msg="Executing migration" id="add share column" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.107342936Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.108152126Z level=info msg="Executing migration" id="create file table" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.109240821Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.109931093Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.110928446Z level=info msg="Executing migration" id="create file_meta table" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.111632074Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.112607839Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.113333282Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.114186448Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-04T15:29:05.384 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.115039993Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.11589141Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.116929598Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.117989839Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.118658492Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.119446406Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.119793351Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.120336735Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.121402997Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.122436991Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.123277576Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.133188659Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.133981654Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.134959781Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.138504352Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-04T15:29:05.385 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.139324288Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.140305527Z level=info msg="Executing migration" id="create folder table" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.141056679Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.142045605Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.142675768Z level=info msg="Executing migration" id="Update folder title length" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.143495897Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=migrator t=2024-03-04T15:29:05.143948501Z level=info msg="migrations completed" performed=485 skipped=0 duration=746.848834ms 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=sqlstore t=2024-03-04T15:29:05.14434483Z level=info msg="Created default organization" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=plugin.loader t=2024-03-04T15:29:05.203593616Z level=info msg="Plugin registered" pluginID=input 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=plugin.loader t=2024-03-04T15:29:05.232888619Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=plugin.loader t=2024-03-04T15:29:05.232906527Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=secrets t=2024-03-04T15:29:05.23293875Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=query_data t=2024-03-04T15:29:05.235070222Z level=info msg="Query Service initialization" 2024-03-04T15:29:05.386 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=live.push_http t=2024-03-04T15:29:05.237620482Z level=info msg="Live Push Gateway initialization" 2024-03-04T15:29:05.511 INFO:teuthology.orchestra.run.smithi053.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-04T15:29:05.511 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300G8480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme1n1 ssd Linux_209ec5e6fafa3475bf02 89.4G Yes 2m ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme2n1 ssd Linux_8a642221cab32acf0129 89.4G Yes 2m ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme3n1 ssd Linux_9ad0515a017afc1b2914 89.4G Yes 2m ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053 /dev/nvme4n1 ssd Linux_aec47f32dfe2106994f3 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 102s ago Has a FileSystem, LVM detected 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi064 /dev/nvme1n1 ssd Linux_40bcd40eb0a93e5d4bdd 89.4G Yes 102s ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi064 /dev/nvme2n1 ssd Linux_6e083438a21556ffed39 89.4G Yes 102s ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi064 /dev/nvme3n1 ssd Linux_6001aadf45c8b55e04a6 89.4G Yes 102s ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi064 /dev/nvme4n1 ssd Linux_85e63a40c15460d9b55a 89.4G No 102s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi160 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JY400BGN 372G No 77s ago Has a FileSystem, LVM detected 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi160 /dev/nvme1n1 ssd Linux_baf30562f5ac9b412ad7 89.4G Yes 77s ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi160 /dev/nvme2n1 ssd Linux_3facdfda0dbef714dd08 89.4G Yes 77s ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi160 /dev/nvme3n1 ssd Linux_f08755519e4a555b6d1d 89.4G Yes 77s ago 2024-03-04T15:29:05.512 INFO:teuthology.orchestra.run.smithi053.stdout:smithi160 /dev/nvme4n1 ssd Linux_820b20529b6356bab516 89.4G No 77s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-04T15:29:05.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:05 smithi053 ceph-mon[25738]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:05.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:05 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:05 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:05 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:05 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:05 smithi053 ceph-mon[25738]: Deploying daemon node-exporter.a on smithi053 2024-03-04T15:29:05.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:05 smithi160 ceph-mon[30717]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:05.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:05 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:05 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:05 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:05 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:05 smithi160 ceph-mon[30717]: Deploying daemon node-exporter.a on smithi053 2024-03-04T15:29:05.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-mon[30297]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:05.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:05.772 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-mon[30297]: Deploying daemon node-exporter.a on smithi053 2024-03-04T15:29:06.148 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=infra.usagestats.collector t=2024-03-04T15:29:05.883812549Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=provisioning.datasources t=2024-03-04T15:29:05.884851566Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=provisioning.datasources t=2024-03-04T15:29:05.896073305Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=provisioning.alerting t=2024-03-04T15:29:05.90721618Z level=info msg="starting to provision alerting" 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=provisioning.alerting t=2024-03-04T15:29:05.90723558Z level=info msg="finished to provision alerting" 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=grafanaStorageLogger t=2024-03-04T15:29:05.907381366Z level=info msg="storage starting" 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=ngalert.state.manager t=2024-03-04T15:29:05.90743451Z level=info msg="Warming state cache for startup" 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=ngalert.state.manager t=2024-03-04T15:29:05.907586306Z level=info msg="State cache has been initialized" states=0 duration=150.851µs 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=ticker t=2024-03-04T15:29:05.907668984Z level=info msg=starting first_tick=2024-03-04T15:29:10Z 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=ngalert.multiorg.alertmanager t=2024-03-04T15:29:05.907702999Z level=info msg="starting MultiOrg Alertmanager" 2024-03-04T15:29:06.272 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=http.server t=2024-03-04T15:29:05.909170176Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-04T15:29:06.273 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:29:05 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a[42843]: logger=sqlstore.transactions t=2024-03-04T15:29:05.929054419Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-04T15:29:06.386 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:06 smithi053 bash[56170]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-04T15:29:06.407 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:29:06.658 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:06 smithi053 ceph-mon[25738]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:06.714 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:06 smithi160 ceph-mon[30717]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:06.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:06 smithi064 ceph-mon[30297]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:29:07.135 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:06 smithi053 bash[56170]: Getting image source signatures 2024-03-04T15:29:07.136 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:06 smithi053 bash[56170]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-04T15:29:07.136 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:06 smithi053 bash[56170]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-04T15:29:07.136 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:06 smithi053 bash[56170]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-04T15:29:07.636 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:07 smithi053 ceph-mon[25738]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:07.714 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:07 smithi160 ceph-mon[30717]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:07.771 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:07 smithi064 ceph-mon[30297]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:08.374 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:08 smithi053 bash[56170]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-04T15:29:08.636 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:08 smithi053 bash[56170]: Writing manifest to image destination 2024-03-04T15:29:09.040 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:08 smithi053 ceph-mon[25738]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:09.040 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:08 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:09.077 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: alertmanager 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: alertmanager 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout:placement: 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: count: 1 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: hosts: 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi053=a 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: created: '2024-03-04T15:28:02.564623Z' 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: ports: 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: - 9093 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: - 9094 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: running: 0 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout: size: 1 2024-03-04T15:29:09.078 INFO:teuthology.orchestra.run.smithi053.stdout:events: 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout:- 2024-03-04T15:28:48.915568Z service:alertmanager [INFO] "service was created" 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout:--- 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: grafana 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: grafana 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout:placement: 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: count: 1 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: hosts: 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi064=a 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout:spec: 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: anonymous_access: true 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: protocol: https 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: created: '2024-03-04T15:28:05.218398Z' 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: ports: 2024-03-04T15:29:09.079 INFO:teuthology.orchestra.run.smithi053.stdout: - 3000 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: running: 0 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: size: 1 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout:events: 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout:- 2024-03-04T15:29:04.429616Z service:grafana [INFO] "service was created" 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout:--- 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: mgr 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: mgr 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout:placement: 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: count: 3 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: hosts: 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi053=y 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi064=x 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi160=z 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.080 INFO:teuthology.orchestra.run.smithi053.stdout: created: '2024-03-04T15:26:26.662752Z' 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: last_refresh: '2024-03-04T15:28:29.448532Z' 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: running: 3 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: size: 3 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout:--- 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: mon 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: mon 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout:placement: 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: count: 3 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: hosts: 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi053:172.21.15.53=a 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi064:172.21.15.64=b 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi160:172.21.15.160=c 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: created: '2024-03-04T15:26:01.167837Z' 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: last_refresh: '2024-03-04T15:28:29.448429Z' 2024-03-04T15:29:09.081 INFO:teuthology.orchestra.run.smithi053.stdout: running: 3 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: size: 3 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout:--- 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: node-exporter 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: node-exporter 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout:placement: 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: count: 3 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: hosts: 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi053=a 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi064=b 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi160=c 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: created: '2024-03-04T15:27:59.797216Z' 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: ports: 2024-03-04T15:29:09.082 INFO:teuthology.orchestra.run.smithi053.stdout: - 9100 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: running: 0 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: size: 3 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout:--- 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: osd 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: osd 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout:unmanaged: true 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout:spec: 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: filter_logic: AND 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: objectstore: bluestore 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: container_image_id: 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: last_refresh: '2024-03-04T15:28:29.448623Z' 2024-03-04T15:29:09.083 INFO:teuthology.orchestra.run.smithi053.stdout: running: 3 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: size: 3 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout:--- 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: prometheus 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: prometheus 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout:placement: 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: count: 1 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: hosts: 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi064=a 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: created: '2024-03-04T15:27:57.058859Z' 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: last_refresh: '2024-03-04T15:28:30.245808Z' 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: ports: 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: - 9095 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: running: 1 2024-03-04T15:29:09.084 INFO:teuthology.orchestra.run.smithi053.stdout: size: 1 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout:--- 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout:service_type: rgw 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout:service_id: foo.a 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout:service_name: rgw.foo.a 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout:placement: 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: count: 1 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: hosts: 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: - smithi053=foo.a 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout:status: 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: created: '2024-03-04T15:27:57.905647Z' 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: last_refresh: '2024-03-04T15:28:35.160444Z' 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: ports: 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: - 80 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: running: 1 2024-03-04T15:29:09.085 INFO:teuthology.orchestra.run.smithi053.stdout: size: 1 2024-03-04T15:29:09.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:08 smithi160 ceph-mon[30717]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:09.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:08 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:09.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:08 smithi064 ceph-mon[30297]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:09.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:08 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:09.385 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 podman[56170]: 2024-03-04 15:29:08.944057466 +0000 UTC m=+2.732088512 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-04T15:29:09.386 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 podman[56170]: 2024-03-04 15:29:09.042421133 +0000 UTC m=+2.830452169 container create b8b2d683682af4fab9dfa60f12f6a74959779031a6d057b0023798e738d4c6a0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:29:09.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-04T15:29:10.002 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:29:10.124 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-mon[25738]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 podman[56170]: 2024-03-04 15:29:09.823618516 +0000 UTC m=+3.611649555 container init b8b2d683682af4fab9dfa60f12f6a74959779031a6d057b0023798e738d4c6a0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 podman[56170]: 2024-03-04 15:29:09.830023165 +0000 UTC m=+3.618054202 container start b8b2d683682af4fab9dfa60f12f6a74959779031a6d057b0023798e738d4c6a0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.835Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.835Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.835Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.835Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.836Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.836Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-04T15:29:10.125 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=arp 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=edac 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-04T15:29:10.126 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=os 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-04T15:29:10.127 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=stat 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=time 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-04T15:29:10.128 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=uname 2024-03-04T15:29:10.129 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-04T15:29:10.129 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-04T15:29:10.129 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.837Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-04T15:29:10.129 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.838Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-04T15:29:10.129 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a[56417]: ts=2024-03-04T15:29:09.838Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-04T15:29:10.129 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 bash[56170]: b8b2d683682af4fab9dfa60f12f6a74959779031a6d057b0023798e738d4c6a0 2024-03-04T15:29:10.129 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:29:09 smithi053 systemd[1]: Started Ceph node-exporter.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:29:10.214 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:09 smithi160 ceph-mon[30717]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-04T15:29:10.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:09 smithi064 ceph-mon[30297]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-04T15:29:11.123 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:10 smithi053 ceph-mon[25738]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:11.123 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:10 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.123 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:10 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.123 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:10 smithi053 ceph-mon[25738]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.123 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:10 smithi053 ceph-mon[25738]: Deploying daemon node-exporter.b on smithi064 2024-03-04T15:29:11.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:10 smithi064 ceph-mon[30297]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:11.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:10 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:10 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:10 smithi064 ceph-mon[30297]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:10 smithi064 ceph-mon[30297]: Deploying daemon node-exporter.b on smithi064 2024-03-04T15:29:11.272 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:11 smithi064 bash[43119]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-04T15:29:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:10 smithi160 ceph-mon[30717]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-04T15:29:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:10 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:10 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:10 smithi160 ceph-mon[30717]: from='mgr.14445 172.21.15.53:0/496656570' entity='mgr.y' 2024-03-04T15:29:11.465 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:10 smithi160 ceph-mon[30717]: Deploying daemon node-exporter.b on smithi064 2024-03-04T15:29:11.865 INFO:teuthology.orchestra.run.smithi053.stdout:osd 3 42s ago - 2024-03-04T15:29:12.412 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ::ffff:172.21.15.64 - - [04/Mar/2024:15:29:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:29:12.413 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-04T15:29:12.423 INFO:tasks.cephadm:Teardown begin 2024-03-04T15:29:12.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:29:12.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:29:12.492 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:29:12.527 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-04T15:29:12.528 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:29:12.542 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:29:12.559 DEBUG:teuthology.orchestra.run.smithi160:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:29:12.585 INFO:tasks.cephadm:Stopping all daemons... 2024-03-04T15:29:12.585 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-04T15:29:12.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a 2024-03-04T15:29:12.877 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:12 smithi053 systemd[1]: Stopping Ceph mon.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:13.132 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a[25714]: 2024-03-04T15:29:12.874+0000 7f4625a2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:13.132 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a[25714]: 2024-03-04T15:29:12.874+0000 7f4625a2d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-04T15:29:13.132 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:12] ENGINE Bus STOPPING 2024-03-04T15:29:13.132 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T15:29:13.132 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:12] ENGINE Bus STOPPED 2024-03-04T15:29:13.132 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:12] ENGINE Bus STARTING 2024-03-04T15:29:13.133 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:12] ENGINE Serving on http://:::9283 2024-03-04T15:29:13.133 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:12 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:12] ENGINE Bus STARTED 2024-03-04T15:29:13.271 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:12 smithi064 bash[43119]: Getting image source signatures 2024-03-04T15:29:13.271 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:12 smithi064 bash[43119]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-04T15:29:13.271 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:12 smithi064 bash[43119]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-04T15:29:13.272 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:12 smithi064 bash[43119]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-04T15:29:13.386 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:13 smithi053 podman[56744]: 2024-03-04 15:29:13.130749369 +0000 UTC m=+0.366059503 container died 4625c153d19876b7caa8b7557dbfbb8b5fc7ae1b5b67627e8d79ed9b4f4e9333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:29:13.674 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:13 smithi053 podman[56744]: 2024-03-04 15:29:13.471911819 +0000 UTC m=+0.707221952 container cleanup 4625c153d19876b7caa8b7557dbfbb8b5fc7ae1b5b67627e8d79ed9b4f4e9333 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-04T15:29:13.674 INFO:journalctl@ceph.mon.a.smithi053.stdout:Mar 04 15:29:13 smithi053 bash[56744]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-a 2024-03-04T15:29:13.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.a.service' 2024-03-04T15:29:13.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:13.934 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-04T15:29:13.934 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-04T15:29:13.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.b 2024-03-04T15:29:14.247 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:13 smithi064 systemd[1]: Stopping Ceph mon.b for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:15.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:15 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-b[30274]: 2024-03-04T15:29:15.150+0000 7f5db2069700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:15.271 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:15 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-b[30274]: 2024-03-04T15:29:15.150+0000 7f5db2069700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-04T15:29:15.272 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:15 smithi064 podman[43213]: 2024-03-04 15:29:15.171439362 +0000 UTC m=+1.057155639 container died 4526efd1013be8b832e1a305f2e6e4ba8ecb28e44145db8c8fb50dd9f2fc6783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-04T15:29:15.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:15] ENGINE Bus STOPPING 2024-03-04T15:29:15.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T15:29:15.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:15] ENGINE Bus STOPPED 2024-03-04T15:29:15.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:15] ENGINE Bus STARTING 2024-03-04T15:29:15.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:15] ENGINE Serving on http://:::9283 2024-03-04T15:29:15.636 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:15 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: [04/Mar/2024:15:29:15] ENGINE Bus STARTED 2024-03-04T15:29:20.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:19 smithi064 podman[43213]: 2024-03-04 15:29:19.577372829 +0000 UTC m=+5.463089105 container cleanup 4526efd1013be8b832e1a305f2e6e4ba8ecb28e44145db8c8fb50dd9f2fc6783 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-b, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True) 2024-03-04T15:29:20.022 INFO:journalctl@ceph.mon.b.smithi064.stdout:Mar 04 15:29:19 smithi064 bash[43213]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-b 2024-03-04T15:29:21.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.b.service' 2024-03-04T15:29:21.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:21.405 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-04T15:29:21.405 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-04T15:29:21.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.c 2024-03-04T15:29:21.521 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:21 smithi064 bash[43119]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-04T15:29:21.522 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:21 smithi064 bash[43119]: Writing manifest to image destination 2024-03-04T15:29:21.715 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:21 smithi160 systemd[1]: Stopping Ceph mon.c for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:22.001 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:21 smithi064 podman[43119]: 2024-03-04 15:29:21.665438939 +0000 UTC m=+10.598179167 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-04T15:29:22.001 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:21 smithi064 podman[43119]: 2024-03-04 15:29:21.75605111 +0000 UTC m=+10.688791368 container create 7f9d2920e44563e072cfe79c06943cd8eeea355af29d0335e6662677df23cb32 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:29:22.001 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:21 smithi064 podman[43119]: 2024-03-04 15:29:21.999596531 +0000 UTC m=+10.932336750 container init 7f9d2920e44563e072cfe79c06943cd8eeea355af29d0335e6662677df23cb32 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:29:22.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:21 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c[30682]: 2024-03-04T15:29:21.778+0000 7fcf30e29700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:22.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:21 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c[30682]: 2024-03-04T15:29:21.778+0000 7fcf30e29700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-04T15:29:22.215 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:21 smithi160 podman[44307]: 2024-03-04 15:29:21.943285906 +0000 UTC m=+0.346140406 container died 19832f78866fd2d4f042dc4c27b9409452d56b1faaaad9dd321d403a07237f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T15:29:22.272 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 podman[43119]: 2024-03-04 15:29:22.005334364 +0000 UTC m=+10.938074584 container start 7f9d2920e44563e072cfe79c06943cd8eeea355af29d0335e6662677df23cb32 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:29:22.272 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.009Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-04T15:29:22.272 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.009Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-04T15:29:22.272 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.009Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-04T15:29:22.272 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.009Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=arp 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=edac 2024-03-04T15:29:22.273 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-04T15:29:22.274 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=os 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=stat 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=time 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-04T15:29:22.275 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=uname 2024-03-04T15:29:22.276 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-04T15:29:22.276 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-04T15:29:22.276 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.010Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-04T15:29:22.276 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.011Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-04T15:29:22.276 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b[43335]: ts=2024-03-04T15:29:22.011Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-04T15:29:22.276 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 bash[43119]: 7f9d2920e44563e072cfe79c06943cd8eeea355af29d0335e6662677df23cb32 2024-03-04T15:29:22.276 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:29:22 smithi064 systemd[1]: Started Ceph node-exporter.b for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:29:22.635 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:22 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: ::ffff:172.21.15.64 - - [04/Mar/2024:15:29:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T15:29:22.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:22 smithi160 podman[44307]: 2024-03-04 15:29:22.401146217 +0000 UTC m=+0.804000720 container cleanup 19832f78866fd2d4f042dc4c27b9409452d56b1faaaad9dd321d403a07237f4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5) 2024-03-04T15:29:22.687 INFO:journalctl@ceph.mon.c.smithi160.stdout:Mar 04 15:29:22 smithi160 bash[44307]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mon-c 2024-03-04T15:29:22.815 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mon.c.service' 2024-03-04T15:29:22.861 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:22.862 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-04T15:29:22.862 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-04T15:29:22.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.y 2024-03-04T15:29:23.313 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:22 smithi053 systemd[1]: Stopping Ceph mgr.y for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:23.602 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:23 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:29:23.310+0000 7f9431fd0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:23.602 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:23 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y[25994]: 2024-03-04T15:29:23.310+0000 7f9431fd0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-04T15:29:23.602 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:23 smithi053 podman[56899]: 2024-03-04 15:29:23.34197872 +0000 UTC m=+0.288579439 container died 5724c6d25cb1eda7e865bf7f73919d14635df5a728a80d3ef92af280edced264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS) 2024-03-04T15:29:23.885 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:23 smithi053 podman[56899]: 2024-03-04 15:29:23.790698232 +0000 UTC m=+0.737299005 container cleanup 5724c6d25cb1eda7e865bf7f73919d14635df5a728a80d3ef92af280edced264 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-03-04T15:29:23.886 INFO:journalctl@ceph.mgr.y.smithi053.stdout:Mar 04 15:29:23 smithi053 bash[56899]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-y 2024-03-04T15:29:24.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-04T15:29:24.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:24.242 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-04T15:29:24.242 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-04T15:29:24.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.x 2024-03-04T15:29:24.569 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:29:24 smithi064 systemd[1]: Stopping Ceph mgr.x for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:24.855 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:29:24 smithi064 podman[43414]: 2024-03-04 15:29:24.56822527 +0000 UTC m=+0.130795479 container died 4fbf03a2016d5f2dde4dc8105c69ba1f94f0c678451ca75c8ca1945a15fb43f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-04T15:29:25.213 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:29:24 smithi064 podman[43414]: 2024-03-04 15:29:24.927083761 +0000 UTC m=+0.489653926 container cleanup 4fbf03a2016d5f2dde4dc8105c69ba1f94f0c678451ca75c8ca1945a15fb43f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T15:29:25.213 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Mar 04 15:29:24 smithi064 bash[43414]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-x 2024-03-04T15:29:25.345 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-04T15:29:25.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:25.381 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-04T15:29:25.381 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-04T15:29:25.381 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.z 2024-03-04T15:29:25.714 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:29:25 smithi160 systemd[1]: Stopping Ceph mgr.z for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:26.049 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:29:25 smithi160 podman[44437]: 2024-03-04 15:29:25.787208144 +0000 UTC m=+0.206363203 container died 89ece9067207bb23fe44014185f691d23794058a86d6397f201db283d9f767c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T15:29:26.390 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:29:26 smithi160 podman[44437]: 2024-03-04 15:29:26.138521762 +0000 UTC m=+0.557676828 container cleanup 89ece9067207bb23fe44014185f691d23794058a86d6397f201db283d9f767c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:29:26.390 INFO:journalctl@ceph.mgr.z.smithi160.stdout:Mar 04 15:29:26 smithi160 bash[44437]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-mgr-z 2024-03-04T15:29:26.564 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@mgr.z.service' 2024-03-04T15:29:26.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:26.599 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-04T15:29:26.600 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-04T15:29:26.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.0 2024-03-04T15:29:26.885 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:26 smithi053 systemd[1]: Stopping Ceph osd.0 for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:27.386 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:26 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0[40429]: 2024-03-04T15:29:26.906+0000 7f3439812700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:27.386 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:26 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0[40429]: 2024-03-04T15:29:26.906+0000 7f3439812700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-04T15:29:27.386 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:26 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0[40429]: 2024-03-04T15:29:26.906+0000 7f3439812700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T15:29:32.673 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:32 smithi053 podman[57032]: 2024-03-04 15:29:32.404019395 +0000 UTC m=+5.619967576 container died bd641edd1c6dc09170f5021aa897e91bb96644ccc809dd3c853610bff2f2d6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=) 2024-03-04T15:29:33.014 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:32 smithi053 podman[57032]: 2024-03-04 15:29:32.837182706 +0000 UTC m=+6.053130883 container cleanup bd641edd1c6dc09170f5021aa897e91bb96644ccc809dd3c853610bff2f2d6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T15:29:33.014 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:32 smithi053 bash[57032]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0 2024-03-04T15:29:33.385 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:33 smithi053 podman[57054]: 2024-03-04 15:29:33.013324752 +0000 UTC m=+0.605645728 container remove bd641edd1c6dc09170f5021aa897e91bb96644ccc809dd3c853610bff2f2d6ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T15:29:33.685 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:33 smithi053 podman[57152]: 2024-03-04 15:29:33.33810809 +0000 UTC m=+0.018407811 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:29:33.685 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:33 smithi053 podman[57152]: 2024-03-04 15:29:33.455665541 +0000 UTC m=+0.135965258 container create 865cb5558d84a27d5419662f09d8720a15e89796a0677ad7322ad7cd8fd731d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-04T15:29:33.685 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:33 smithi053 podman[57152]: 2024-03-04 15:29:33.683636854 +0000 UTC m=+0.363936569 container init 865cb5558d84a27d5419662f09d8720a15e89796a0677ad7322ad7cd8fd731d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=) 2024-03-04T15:29:33.989 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:33 smithi053 podman[57152]: 2024-03-04 15:29:33.689924878 +0000 UTC m=+0.370224596 container start 865cb5558d84a27d5419662f09d8720a15e89796a0677ad7322ad7cd8fd731d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T15:29:33.989 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:33 smithi053 podman[57152]: 2024-03-04 15:29:33.782685112 +0000 UTC m=+0.462984839 container attach 865cb5558d84a27d5419662f09d8720a15e89796a0677ad7322ad7cd8fd731d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-04T15:29:34.286 INFO:journalctl@ceph.osd.0.smithi053.stdout:Mar 04 15:29:33 smithi053 podman[57152]: 2024-03-04 15:29:33.987821063 +0000 UTC m=+0.668120786 container died 865cb5558d84a27d5419662f09d8720a15e89796a0677ad7322ad7cd8fd731d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-0-deactivate, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-04T15:29:34.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.0.service' 2024-03-04T15:29:34.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:34.634 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-04T15:29:34.634 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-04T15:29:34.634 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.1 2024-03-04T15:29:34.935 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:34 smithi064 systemd[1]: Stopping Ceph osd.1 for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:35.271 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:34 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1[34871]: 2024-03-04T15:29:34.933+0000 7fb352a0c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:35.272 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:34 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1[34871]: 2024-03-04T15:29:34.933+0000 7fb352a0c700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-04T15:29:35.272 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:34 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1[34871]: 2024-03-04T15:29:34.933+0000 7fb352a0c700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T15:29:40.642 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:40 smithi064 podman[43560]: 2024-03-04 15:29:40.363589377 +0000 UTC m=+5.533954717 container died 372d4b963d5e5f1a59a42d1a5aad385cec874dd2f7bc48d602efba957c54743c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-04T15:29:41.021 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:40 smithi064 podman[43560]: 2024-03-04 15:29:40.756732429 +0000 UTC m=+5.927097773 container cleanup 372d4b963d5e5f1a59a42d1a5aad385cec874dd2f7bc48d602efba957c54743c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:29:41.021 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:40 smithi064 bash[43560]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1 2024-03-04T15:29:41.327 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:41 smithi064 podman[43582]: 2024-03-04 15:29:41.024197954 +0000 UTC m=+0.655422300 container remove 372d4b963d5e5f1a59a42d1a5aad385cec874dd2f7bc48d602efba957c54743c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T15:29:41.628 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:41 smithi064 podman[43678]: 2024-03-04 15:29:41.228890789 +0000 UTC m=+0.017969531 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:29:41.629 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:41 smithi064 podman[43678]: 2024-03-04 15:29:41.333424668 +0000 UTC m=+0.122503446 container create 5470cac90bba7688110ae2e138310be95d801db55e6d48a72bffb1a44fa2cb2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS) 2024-03-04T15:29:41.629 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:41 smithi064 podman[43678]: 2024-03-04 15:29:41.56094873 +0000 UTC m=+0.350027468 container init 5470cac90bba7688110ae2e138310be95d801db55e6d48a72bffb1a44fa2cb2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-04T15:29:41.629 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:41 smithi064 podman[43678]: 2024-03-04 15:29:41.566693841 +0000 UTC m=+0.355772587 container start 5470cac90bba7688110ae2e138310be95d801db55e6d48a72bffb1a44fa2cb2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-04T15:29:41.629 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:41 smithi064 podman[43678]: 2024-03-04 15:29:41.627174395 +0000 UTC m=+0.416253149 container attach 5470cac90bba7688110ae2e138310be95d801db55e6d48a72bffb1a44fa2cb2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-04T15:29:42.021 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:41 smithi064 podman[43678]: 2024-03-04 15:29:41.864800158 +0000 UTC m=+0.653878893 container died 5470cac90bba7688110ae2e138310be95d801db55e6d48a72bffb1a44fa2cb2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T15:29:42.354 INFO:journalctl@ceph.osd.1.smithi064.stdout:Mar 04 15:29:42 smithi064 podman[43724]: 2024-03-04 15:29:42.338616591 +0000 UTC m=+0.463582269 container remove 5470cac90bba7688110ae2e138310be95d801db55e6d48a72bffb1a44fa2cb2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-04T15:29:42.359 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.1.service' 2024-03-04T15:29:42.400 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:42.400 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-04T15:29:42.400 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-04T15:29:42.400 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.2 2024-03-04T15:29:42.715 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:42 smithi160 systemd[1]: Stopping Ceph osd.2 for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:43.215 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:42 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2[35980]: 2024-03-04T15:29:42.717+0000 7fc0d9a9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:43.215 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:42 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2[35980]: 2024-03-04T15:29:42.717+0000 7fc0d9a9d700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-04T15:29:43.215 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:42 smithi160 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2[35980]: 2024-03-04T15:29:42.717+0000 7fc0d9a9d700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T15:29:48.459 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:48 smithi160 podman[44581]: 2024-03-04 15:29:48.155994767 +0000 UTC m=+5.552904327 container died d525123c8d9b441ae18dfdc83471bf69b24d1101bda3e335e9f2e272ad5c4724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-04T15:29:48.715 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:48 smithi160 podman[44581]: 2024-03-04 15:29:48.531442068 +0000 UTC m=+5.928351636 container cleanup d525123c8d9b441ae18dfdc83471bf69b24d1101bda3e335e9f2e272ad5c4724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-04T15:29:48.715 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:48 smithi160 bash[44581]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2 2024-03-04T15:29:48.715 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:48 smithi160 podman[44603]: 2024-03-04 15:29:48.665554719 +0000 UTC m=+0.512431819 container remove d525123c8d9b441ae18dfdc83471bf69b24d1101bda3e335e9f2e272ad5c4724 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-04T15:29:49.361 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:49 smithi160 podman[44699]: 2024-03-04 15:29:48.965486951 +0000 UTC m=+0.017507691 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:29:49.361 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:49 smithi160 podman[44699]: 2024-03-04 15:29:49.066207856 +0000 UTC m=+0.118228622 container create bca42997109bc201bb1f05c084650289fdf1d0fb605460c80b774998ed0b6676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-04T15:29:49.361 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:49 smithi160 podman[44699]: 2024-03-04 15:29:49.268518888 +0000 UTC m=+0.320539616 container init bca42997109bc201bb1f05c084650289fdf1d0fb605460c80b774998ed0b6676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-04T15:29:49.362 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:49 smithi160 podman[44699]: 2024-03-04 15:29:49.274391187 +0000 UTC m=+0.326411914 container start bca42997109bc201bb1f05c084650289fdf1d0fb605460c80b774998ed0b6676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:29:49.362 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:49 smithi160 podman[44699]: 2024-03-04 15:29:49.360068529 +0000 UTC m=+0.412089258 container attach bca42997109bc201bb1f05c084650289fdf1d0fb605460c80b774998ed0b6676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T15:29:49.715 INFO:journalctl@ceph.osd.2.smithi160.stdout:Mar 04 15:29:49 smithi160 podman[44699]: 2024-03-04 15:29:49.570993666 +0000 UTC m=+0.623014386 container died bca42997109bc201bb1f05c084650289fdf1d0fb605460c80b774998ed0b6676 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-osd-2-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-04T15:29:50.141 DEBUG:teuthology.orchestra.run.smithi160:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@osd.2.service' 2024-03-04T15:29:50.182 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:29:50.183 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-04T15:29:50.183 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-04T15:29:50.183 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@rgw.foo.a 2024-03-04T15:29:50.487 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:29:50 smithi053 systemd[1]: Stopping Ceph rgw.foo.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:29:50.886 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:29:50 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a[47220]: 2024-03-04T15:29:50.484+0000 7f193e74a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:29:50.886 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:29:50 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a[47220]: 2024-03-04T15:29:50.484+0000 7f194cfc2ac0 -1 shutting down 2024-03-04T15:30:00.886 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:30:00 smithi053 bash[57285]: time="2024-03-04T15:30:00Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-04T15:30:00.886 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:30:00 smithi053 podman[57285]: 2024-03-04 15:30:00.509492619 +0000 UTC m=+10.133884998 container died 0062d4ed96bdba4a62b295c0aba669af15bb18b6dc0f4e6fd4d291f2b085aa3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-04T15:30:01.351 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:30:01 smithi053 podman[57285]: 2024-03-04 15:30:01.081941984 +0000 UTC m=+10.706334371 container cleanup 0062d4ed96bdba4a62b295c0aba669af15bb18b6dc0f4e6fd4d291f2b085aa3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:30:01.351 INFO:journalctl@ceph.rgw.foo.a.smithi053.stdout:Mar 04 15:30:01 smithi053 bash[57285]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-rgw-foo-a 2024-03-04T15:30:01.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@rgw.foo.a.service' 2024-03-04T15:30:01.552 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:30:01.553 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-04T15:30:01.553 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-04T15:30:01.553 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@prometheus.a 2024-03-04T15:30:01.771 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 systemd[1]: Stopping Ceph prometheus.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.883Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-04T15:30:02.234 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a[40965]: ts=2024-03-04T15:30:01.884Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-04T15:30:02.235 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:01 smithi064 podman[43810]: 2024-03-04 15:30:01.894312859 +0000 UTC m=+0.146530726 container died 71e0e568b872a616cf667687e5275b27b15937d5459626901208abe054fffd02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:02.521 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:02 smithi064 podman[43810]: 2024-03-04 15:30:02.338957857 +0000 UTC m=+0.591175718 container cleanup 71e0e568b872a616cf667687e5275b27b15937d5459626901208abe054fffd02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:02.521 INFO:journalctl@ceph.prometheus.a.smithi064.stdout:Mar 04 15:30:02 smithi064 bash[43810]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T15:30:02.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@prometheus.a.service' 2024-03-04T15:30:02.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T15:30:02.797 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-04T15:30:02.797 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T15:30:02.993 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:30:04.253 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:03 smithi053 systemd[1]: Stopping Ceph alertmanager.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:30:04.253 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:04 smithi053 ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a[54539]: ts=2024-03-04T15:30:04.230Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-04T15:30:04.570 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:04 smithi053 podman[57642]: 2024-03-04 15:30:04.251574486 +0000 UTC m=+0.122143541 container died 96b844979cbdcf4d4e61e1e5201548242e0b4e268106f111e1d4ff28bafecfc1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:04.570 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:04 smithi053 podman[57642]: 2024-03-04 15:30:04.569573128 +0000 UTC m=+0.440142182 container cleanup 96b844979cbdcf4d4e61e1e5201548242e0b4e268106f111e1d4ff28bafecfc1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:04.847 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:04 smithi053 bash[57642]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a 2024-03-04T15:30:04.847 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:04 smithi053 podman[57661]: 2024-03-04 15:30:04.762125765 +0000 UTC m=+0.509164112 container remove 96b844979cbdcf4d4e61e1e5201548242e0b4e268106f111e1d4ff28bafecfc1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:04.847 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:04 smithi053 podman[57661]: 2024-03-04 15:30:04.846118265 +0000 UTC m=+0.593156619 volume remove 9e3235ddcb83ef24e4c8482265274cebe0f965d975b4fbedb55127aefb661b5c 2024-03-04T15:30:05.632 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:05 smithi053 systemd[1]: Stopping Ceph node-exporter.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:30:05.632 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:05 smithi053 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-04T15:30:05.633 INFO:journalctl@ceph.alertmanager.a.smithi053.stdout:Mar 04 15:30:05 smithi053 systemd[1]: Stopped Ceph alertmanager.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:30:05.886 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:05 smithi053 podman[57777]: 2024-03-04 15:30:05.628237781 +0000 UTC m=+0.128548157 container died b8b2d683682af4fab9dfa60f12f6a74959779031a6d057b0023798e738d4c6a0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:06.334 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:06 smithi053 podman[57777]: 2024-03-04 15:30:06.081617205 +0000 UTC m=+0.581927617 container cleanup b8b2d683682af4fab9dfa60f12f6a74959779031a6d057b0023798e738d4c6a0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:06.334 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:06 smithi053 bash[57777]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a 2024-03-04T15:30:06.611 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:06 smithi053 podman[57796]: 2024-03-04 15:30:06.332930717 +0000 UTC m=+0.705257950 container remove b8b2d683682af4fab9dfa60f12f6a74959779031a6d057b0023798e738d4c6a0 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T15:30:06.611 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:06 smithi053 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:30:06.612 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:06 smithi053 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-04T15:30:06.612 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:06 smithi053 systemd[1]: Stopped Ceph node-exporter.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:30:06.612 INFO:journalctl@ceph.node-exporter.a.smithi053.stdout:Mar 04 15:30:06 smithi053 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.a.service: Consumed 1.774s CPU time. 2024-03-04T15:30:07.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T15:30:07.281 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:30:08.271 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:08 smithi064 systemd[1]: Stopping Ceph grafana.a for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:30:08.645 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:08 smithi064 podman[44160]: 2024-03-04 15:30:08.372920849 +0000 UTC m=+0.101203377 container died 05b263358208d82bebd4d3afcb741fd762b42ba9f818e7d4af0963b6391c6864 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a, maintainer=Paul Cuzner , description=Ceph Grafana Container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, distribution-scope=public, architecture=x86_64, vcs-type=git, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=) 2024-03-04T15:30:08.646 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:08 smithi064 podman[44160]: 2024-03-04 15:30:08.617305971 +0000 UTC m=+0.345588494 container cleanup 05b263358208d82bebd4d3afcb741fd762b42ba9f818e7d4af0963b6391c6864 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a, io.openshift.expose-services=, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, vcs-type=git, build-date=2022-04-29T11:48:11.055897) 2024-03-04T15:30:08.646 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:08 smithi064 bash[44160]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a 2024-03-04T15:30:09.066 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:08 smithi064 podman[44179]: 2024-03-04 15:30:08.81022794 +0000 UTC m=+0.435947283 container remove 05b263358208d82bebd4d3afcb741fd762b42ba9f818e7d4af0963b6391c6864 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-grafana-a, io.openshift.expose-services=, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, maintainer=Paul Cuzner , vcs-type=git, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8) 2024-03-04T15:30:09.066 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:08 smithi064 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:30:09.067 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:09 smithi064 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-04T15:30:09.067 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:09 smithi064 systemd[1]: Stopped Ceph grafana.a for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:30:09.067 INFO:journalctl@ceph.grafana.a.smithi064.stdout:Mar 04 15:30:09 smithi064 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@grafana.a.service: Consumed 2.779s CPU time. 2024-03-04T15:30:09.502 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:09 smithi064 systemd[1]: Stopping Ceph node-exporter.b for 1fe97ada-da3b-11ee-95c6-87774f69a715... 2024-03-04T15:30:09.771 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:09 smithi064 podman[44297]: 2024-03-04 15:30:09.50135979 +0000 UTC m=+0.118055290 container died 7f9d2920e44563e072cfe79c06943cd8eeea355af29d0335e6662677df23cb32 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:30:10.250 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:09 smithi064 podman[44297]: 2024-03-04 15:30:09.837852072 +0000 UTC m=+0.454547585 container cleanup 7f9d2920e44563e072cfe79c06943cd8eeea355af29d0335e6662677df23cb32 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:30:10.250 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:09 smithi064 bash[44297]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b 2024-03-04T15:30:10.250 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:10 smithi064 podman[44316]: 2024-03-04 15:30:10.030277382 +0000 UTC m=+0.532354593 container remove 7f9d2920e44563e072cfe79c06943cd8eeea355af29d0335e6662677df23cb32 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-1fe97ada-da3b-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T15:30:10.250 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:10 smithi064 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-04T15:30:10.521 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:10 smithi064 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-04T15:30:10.521 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:10 smithi064 systemd[1]: Stopped Ceph node-exporter.b for 1fe97ada-da3b-11ee-95c6-87774f69a715. 2024-03-04T15:30:10.521 INFO:journalctl@ceph.node-exporter.b.smithi064.stdout:Mar 04 15:30:10 smithi064 systemd[1]: ceph-1fe97ada-da3b-11ee-95c6-87774f69a715@node-exporter.b.service: Consumed 1.795s CPU time. 2024-03-04T15:30:10.907 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T15:30:11.099 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:30:12.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:30:12.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:30:12.320 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:30:12.349 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-04T15:30:12.351 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi053/crash 2024-03-04T15:30:12.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash -- . 2024-03-04T15:30:12.391 INFO:teuthology.orchestra.run.smithi053.stderr:tar: /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T15:30:12.392 INFO:teuthology.orchestra.run.smithi053.stderr:tar: Error is not recoverable: exiting now 2024-03-04T15:30:12.393 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi064/crash 2024-03-04T15:30:12.394 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash -- . 2024-03-04T15:30:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T15:30:12.446 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-03-04T15:30:12.448 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi160/crash 2024-03-04T15:30:12.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash -- . 2024-03-04T15:30:12.497 INFO:teuthology.orchestra.run.smithi160.stderr:tar: /var/lib/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T15:30:12.497 INFO:teuthology.orchestra.run.smithi160.stderr:tar: Error is not recoverable: exiting now 2024-03-04T15:30:12.498 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-04T15:30:12.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:30:12.530 INFO:teuthology.orchestra.run.smithi053.stdout:2024-03-04T15:26:19.907133+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:30:12.531 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-04T15:30:12.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[SEC\]' /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:30:12.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]' /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:30:12.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[WRN\]' /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-04T15:30:12.741 INFO:teuthology.orchestra.run.smithi053.stdout:2024-03-04T15:26:19.907133+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-04T15:30:12.741 INFO:tasks.cephadm:Compressing logs... 2024-03-04T15:30:12.741 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T15:30:12.783 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T15:30:12.786 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T15:30:12.808 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T15:30:12.808 INFO:teuthology.orchestra.run.smithi053.stderr:: No such file or directory 2024-03-04T15:30:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-04T15:30:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T15:30:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-04T15:30:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-04T15:30:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-04T15:30:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.y.log: 57.8% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-04T15:30:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T15:30:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log 2024-03-04T15:30:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log: 89.3% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T15:30:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T15:30:12.812 INFO:teuthology.orchestra.run.smithi053.stderr: 90.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T15:30:12.812 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log: 82.7% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T15:30:12.812 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log: 84.6% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T15:30:12.812 INFO:teuthology.orchestra.run.smithi160.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T15:30:12.812 INFO:teuthology.orchestra.run.smithi160.stderr:: No such file or directory 2024-03-04T15:30:12.813 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T15:30:12.814 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.z.log 2024-03-04T15:30:12.814 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T15:30:12.814 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log: /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log 2024-03-04T15:30:12.814 INFO:teuthology.orchestra.run.smithi160.stderr: 90.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T15:30:12.815 INFO:teuthology.orchestra.run.smithi053.stderr: 90.5% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T15:30:12.817 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-04T15:30:12.817 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log: /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T15:30:12.817 INFO:teuthology.orchestra.run.smithi160.stderr: 81.3% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T15:30:12.817 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.2.log: gzip -5 89.5% --verbose -- -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T15:30:12.817 INFO:teuthology.orchestra.run.smithi160.stderr: /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-04T15:30:12.817 INFO:teuthology.orchestra.run.smithi160.stderr: 84.2% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T15:30:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T15:30:12.818 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr: 89.5% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr: 81.4% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr: 81.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T15:30:12.819 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log: 83.9% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T15:30:12.820 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log: /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.c.log: 90.2% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.z.log.gz 2024-03-04T15:30:12.820 INFO:teuthology.orchestra.run.smithi160.stderr: 90.5% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T15:30:12.820 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.b.log: 90.2% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-04T15:30:12.822 INFO:teuthology.orchestra.run.smithi064.stderr: 90.5% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T15:30:12.868 INFO:teuthology.orchestra.run.smithi160.stderr: 92.2% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-04T15:30:12.871 INFO:teuthology.orchestra.run.smithi064.stderr: 92.2% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-04T15:30:12.876 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-04T15:30:12.984 INFO:teuthology.orchestra.run.smithi053.stderr: 91.3% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-04T15:30:13.070 INFO:teuthology.orchestra.run.smithi160.stderr: 94.2% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-04T15:30:13.072 INFO:teuthology.orchestra.run.smithi160.stderr: 2024-03-04T15:30:13.072 INFO:teuthology.orchestra.run.smithi160.stderr:real 0m0.272s 2024-03-04T15:30:13.072 INFO:teuthology.orchestra.run.smithi160.stderr:user 0m0.314s 2024-03-04T15:30:13.072 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m0.037s 2024-03-04T15:30:13.073 INFO:teuthology.orchestra.run.smithi053.stderr: 94.3% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-04T15:30:13.075 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-04T15:30:13.076 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m0.280s 2024-03-04T15:30:13.076 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m0.507s 2024-03-04T15:30:13.076 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.039s 2024-03-04T15:30:13.136 INFO:teuthology.orchestra.run.smithi064.stderr: 94.2% -- replaced with /var/log/ceph/1fe97ada-da3b-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-04T15:30:13.138 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-03-04T15:30:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.338s 2024-03-04T15:30:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m0.393s 2024-03-04T15:30:13.138 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.031s 2024-03-04T15:30:13.139 INFO:tasks.cephadm:Archiving logs... 2024-03-04T15:30:13.139 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi053/log 2024-03-04T15:30:13.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T15:30:13.260 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi064/log 2024-03-04T15:30:13.261 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T15:30:13.351 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi160/log 2024-03-04T15:30:13.352 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T15:30:13.438 INFO:tasks.cephadm:Removing cluster... 2024-03-04T15:30:13.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 --force 2024-03-04T15:30:13.622 INFO:teuthology.orchestra.run.smithi053.stdout:Deleting cluster with fsid: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:30:13.737 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 --force 2024-03-04T15:30:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:30:14.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1fe97ada-da3b-11ee-95c6-87774f69a715 --force 2024-03-04T15:30:14.227 INFO:teuthology.orchestra.run.smithi160.stdout:Deleting cluster with fsid: 1fe97ada-da3b-11ee-95c6-87774f69a715 2024-03-04T15:30:14.341 INFO:tasks.cephadm:Removing cephadm ... 2024-03-04T15:30:14.342 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T15:30:14.358 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T15:30:14.373 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T15:30:14.388 INFO:tasks.cephadm:Teardown complete 2024-03-04T15:30:14.388 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-04T15:30:14.399 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi053:/dev/vg_nvme/lv_1... 2024-03-04T15:30:14.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo nvme disconnect -n lv_1 2024-03-04T15:30:14.551 INFO:teuthology.orchestra.run.smithi053.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-04T15:30:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:14.552 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi053:/dev/vg_nvme/lv_2... 2024-03-04T15:30:14.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo nvme disconnect -n lv_2 2024-03-04T15:30:14.691 INFO:teuthology.orchestra.run.smithi053.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-04T15:30:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:14.693 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi053:/dev/vg_nvme/lv_3... 2024-03-04T15:30:14.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo nvme disconnect -n lv_3 2024-03-04T15:30:14.872 INFO:teuthology.orchestra.run.smithi053.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-04T15:30:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:14.874 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi053:/dev/vg_nvme/lv_4... 2024-03-04T15:30:14.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo nvme disconnect -n lv_4 2024-03-04T15:30:15.061 INFO:teuthology.orchestra.run.smithi053.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-04T15:30:15.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:15.063 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-04T15:30:15.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/scratch_devs 2024-03-04T15:30:15.090 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_1... 2024-03-04T15:30:15.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_1 2024-03-04T15:30:15.255 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-04T15:30:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:15.256 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_2... 2024-03-04T15:30:15.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_2 2024-03-04T15:30:15.469 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-04T15:30:15.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:15.471 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_3... 2024-03-04T15:30:15.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_3 2024-03-04T15:30:15.627 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-04T15:30:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:15.628 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_4... 2024-03-04T15:30:15.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_4 2024-03-04T15:30:15.812 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-04T15:30:15.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:15.813 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-03-04T15:30:15.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/scratch_devs 2024-03-04T15:30:15.841 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi160:/dev/vg_nvme/lv_1... 2024-03-04T15:30:15.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo nvme disconnect -n lv_1 2024-03-04T15:30:15.987 INFO:teuthology.orchestra.run.smithi160.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-04T15:30:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:15.989 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi160:/dev/vg_nvme/lv_2... 2024-03-04T15:30:15.989 DEBUG:teuthology.orchestra.run.smithi160:> sudo nvme disconnect -n lv_2 2024-03-04T15:30:16.127 INFO:teuthology.orchestra.run.smithi160.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-04T15:30:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:16.129 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi160:/dev/vg_nvme/lv_3... 2024-03-04T15:30:16.129 DEBUG:teuthology.orchestra.run.smithi160:> sudo nvme disconnect -n lv_3 2024-03-04T15:30:16.264 INFO:teuthology.orchestra.run.smithi160.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-04T15:30:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:16.265 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi160:/dev/vg_nvme/lv_4... 2024-03-04T15:30:16.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo nvme disconnect -n lv_4 2024-03-04T15:30:16.406 INFO:teuthology.orchestra.run.smithi160.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-04T15:30:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:16.408 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-03-04T15:30:16.408 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/scratch_devs 2024-03-04T15:30:16.439 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T15:30:16.450 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T15:30:16.451 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:30:16.457 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:30:16.461 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:30:16.470 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:30:16.472 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-03-04T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 -4020us[-4020us] +/- 91ms 2024-03-04T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -844us[ -782us] +/- 60ms 2024-03-04T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +4251us[+4313us] +/- 73ms 2024-03-04T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:30:16.475 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:30:16.475 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-03-04T15:30:16.475 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 -4034us[-3923us] +/- 91ms 2024-03-04T15:30:16.476 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 32 -876us[ -765us] +/- 60ms 2024-03-04T15:30:16.476 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +4204us[+4204us] +/- 73ms 2024-03-04T15:30:16.476 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:30:16.494 INFO:teuthology.orchestra.run.smithi160.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:30:16.500 INFO:teuthology.orchestra.run.smithi160.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:30:16.500 INFO:teuthology.orchestra.run.smithi160.stdout:=============================================================================== 2024-03-04T15:30:16.500 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 -4025us[-4025us] +/- 91ms 2024-03-04T15:30:16.500 INFO:teuthology.orchestra.run.smithi160.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -841us[ -790us] +/- 60ms 2024-03-04T15:30:16.501 INFO:teuthology.orchestra.run.smithi160.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +4265us[+4265us] +/- 73ms 2024-03-04T15:30:16.501 INFO:teuthology.orchestra.run.smithi160.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:30:16.501 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T15:30:16.511 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T15:30:16.511 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T15:30:16.520 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T15:30:16.596 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T15:30:16.664 DEBUG:teuthology.orchestra.run.smithi160:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T15:30:16.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:30:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:16.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:30:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:16.798 DEBUG:teuthology.orchestra.run.smithi160:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:30:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:16.830 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T15:30:16.840 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T15:30:16.848 INFO:teuthology.task.internal:Duration was 703.029135 seconds 2024-03-04T15:30:16.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T15:30:16.857 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T15:30:16.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T15:30:16.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T15:30:16.865 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T15:30:16.902 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:30:16.905 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:30:16.914 INFO:teuthology.orchestra.run.smithi160.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:30:17.309 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T15:30:17.309 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-03-04T15:30:17.309 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T15:30:17.334 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-03-04T15:30:17.334 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T15:30:17.374 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2024-03-04T15:30:17.375 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T15:30:17.406 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T15:30:17.406 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T15:30:17.414 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T15:30:17.423 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T15:30:17.485 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T15:30:17.485 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T15:30:17.493 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T15:30:17.508 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T15:30:17.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T15:30:17.722 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T15:30:17.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T15:30:17.755 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T15:30:17.785 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T15:30:17.813 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T15:30:17.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:30:17.846 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:30:17.854 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:30:17.868 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-03-04T15:30:17.877 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-03-04T15:30:17.884 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2024-03-04T15:30:17.912 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:30:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:17.972 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:30:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:17.991 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T15:30:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:30:18.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T15:30:18.017 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T15:30:18.018 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi053 2024-03-04T15:30:18.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T15:30:18.069 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi064 2024-03-04T15:30:18.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T15:30:18.115 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170/remote/smithi160 2024-03-04T15:30:18.116 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T15:30:18.169 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T15:30:18.169 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T15:30:18.173 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T15:30:18.177 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T15:30:18.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T15:30:18.252 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T15:30:18.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T15:30:18.276 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T15:30:18.277 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T15:30:18.285 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T15:30:18.288 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T15:30:18.296 INFO:teuthology.orchestra.run.smithi053.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 15:30 /home/ubuntu/cephtest 2024-03-04T15:30:18.300 INFO:teuthology.orchestra.run.smithi064.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 15:30 /home/ubuntu/cephtest 2024-03-04T15:30:18.302 INFO:teuthology.orchestra.run.smithi160.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 15:30 /home/ubuntu/cephtest 2024-03-04T15:30:18.303 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T15:30:18.325 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T15:30:18.405 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T15:30:18.443 DEBUG:teuthology.nuke:shortname: smithi053 2024-03-04T15:30:18.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:30:18.490 DEBUG:teuthology.nuke:shortname: smithi064 2024-03-04T15:30:18.490 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:30:18.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:08:57.467354', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAtfG8/IeHVzu1kUVyWoGuv8wn8we0PiWl4Kdr7CreaPLybqYO9k3EL0GF557B7CYgh35UBoTsav0uW1y4Rvct8='} 2024-03-04T15:30:18.532 DEBUG:teuthology.nuke:shortname: smithi160 2024-03-04T15:30:18.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:30:18.548 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:08:57.469301', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKIQpmCis09si6l5fmDFPlVYx2CB2vjFirqGOyXiKNVGookbxjrDHPn21w/9SWtyGQu31tsgXKqiuOOTtJvOJvU='} 2024-03-04T15:30:18.574 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581170', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:08:57.470422', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPFOHYWIpXU+ozeWSPqmBmSb9SYag2Eyafp0sZCWWf2UTe8dQXltW6tzct8gGYZm/i1Tn2jOGr/gJNm7FXDTc5M='} 2024-03-04T15:30:18.614 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-04T15:30:18.614 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T15:30:18.661 INFO:teuthology.orchestra.console.smithi160:Power off 2024-03-04T15:30:18.661 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T15:30:18.677 INFO:teuthology.orchestra.console.smithi064:Power off 2024-03-04T15:30:18.677 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T15:30:18.689 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-04T15:30:18.689 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:18.717 DEBUG:teuthology.orchestra.console.smithi160:power off output: Chassis Power Control: Down/Off 2024-03-04T15:30:18.718 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:18.744 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-03-04T15:30:18.745 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:18.753 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-04T15:30:18.768 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2024-03-04T15:30:18.793 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-04T15:30:22.755 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:22.769 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:22.794 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:22.865 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-04T15:30:22.878 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is on 2024-03-04T15:30:22.904 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-03-04T15:30:26.868 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:26.878 DEBUG:teuthology.orchestra.console.smithi160:pexpect command: ipmitool -H smithi160.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:26.905 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T15:30:26.979 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-04T15:30:26.988 DEBUG:teuthology.orchestra.console.smithi160:check power output: Chassis Power is off 2024-03-04T15:30:27.015 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-03-04T15:30:27.081 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-04T15:30:27.089 INFO:teuthology.orchestra.console.smithi160:Power off completed 2024-03-04T15:30:27.116 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-03-04T15:30:27.220 INFO:teuthology.run:Summary data: description: orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} duration: 703.0291347503662 failure_reason: '"2024-03-04T15:26:19.907133+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_tonay@teuthology success: false 2024-03-04T15:30:27.221 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:30:27.311 INFO:teuthology.run:FAIL