2024-05-15T12:22:39.770 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-15T12:22:39.770 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113 --verbose --owner scheduled_rzarzynski@teuthology --name rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi --description rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} -- /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113/orig.config.yaml 2024-05-15T12:22:39.797 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T12:22:39.877 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113 branch: wip-osd-ec-partial-reads description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: null first_in_suite: false job_id: '7707113' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-osd-ec-partial-reads ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-osd-ec-partial-reads sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 owner: scheduled_rzarzynski@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8148 sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-osd-ec-partial-reads suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDIUIZBwE779FyEaMeCCDgGFEeYdaCbVUSbENGRifPeaJI22p9QmvYPLQlhTR9D3rHd9ZKj17GEARvZcfff2l1U= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALcqEcbx3n5PXcdBC5gw5wam9JF2wZLtYfPORwQyGP2YFQEzWUPOYocinNfdL4GITk08daNGkTWuxOQTCGeAPI= tasks: - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-15_11:53:37 tube: smithi user: rzarzynski verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-15T12:22:39.877 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa; will attempt to use it 2024-05-15T12:22:39.878 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ae2481253f9b2fa3451954e615eedb67ebdd5314/qa/tasks 2024-05-15T12:22:39.878 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-15T12:22:39.880 INFO:teuthology.task.internal:Checking packages... 2024-05-15T12:22:39.910 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ae2481253f9b2fa3451954e615eedb67ebdd5314' 2024-05-15T12:22:39.910 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-15T12:22:39.910 INFO:teuthology.packaging:ref: None 2024-05-15T12:22:39.910 INFO:teuthology.packaging:tag: None 2024-05-15T12:22:39.910 INFO:teuthology.packaging:branch: wip-osd-ec-partial-reads 2024-05-15T12:22:39.910 INFO:teuthology.packaging:sha1: ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:22:39.910 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-osd-ec-partial-reads 2024-05-15T12:22:40.152 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1882.gae248125 2024-05-15T12:22:40.153 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-15T12:22:40.160 INFO:teuthology.task.internal:no buildpackages task found 2024-05-15T12:22:40.160 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-15T12:22:40.204 INFO:teuthology.task.internal:Saving configuration 2024-05-15T12:22:40.217 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-15T12:22:40.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-15T12:22:40.261 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:46.103052', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDIUIZBwE779FyEaMeCCDgGFEeYdaCbVUSbENGRifPeaJI22p9QmvYPLQlhTR9D3rHd9ZKj17GEARvZcfff2l1U='} 2024-05-15T12:22:40.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-15 12:17:46.105218', 'locked_by': 'scheduled_rzarzynski@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALcqEcbx3n5PXcdBC5gw5wam9JF2wZLtYfPORwQyGP2YFQEzWUPOYocinNfdL4GITk08daNGkTWuxOQTCGeAPI='} 2024-05-15T12:22:40.287 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-15T12:22:40.293 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-05-15T12:22:40.293 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-05-15T12:22:40.293 INFO:teuthology.run_tasks:Running task console_log... 2024-05-15T12:22:40.381 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f73dd259160>, signals=[15]) 2024-05-15T12:22:40.382 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-15T12:22:40.390 INFO:teuthology.task.internal:Opening connections... 2024-05-15T12:22:40.390 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-05-15T12:22:40.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:22:40.469 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-05-15T12:22:40.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:22:40.546 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-15T12:22:40.552 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-05-15T12:22:40.570 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-05-15T12:22:40.570 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:22:40.627 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:22:40.628 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-15T12:22:40.628 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:22:40.628 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-05-15T12:22:40.649 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-05-15T12:22:40.666 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-05-15T12:22:40.667 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-05-15T12:22:40.725 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-05-15T12:22:40.725 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-05-15T12:22:40.725 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-15T12:22:40.726 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-15T12:22:40.727 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-05-15T12:22:40.749 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-15T12:22:40.758 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-15T12:22:40.779 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-15T12:22:40.780 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:22:40.782 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-05-15T12:22:40.799 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-15T12:22:40.803 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-15T12:22:40.803 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:22:40.838 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-05-15T12:22:40.867 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-15T12:22:40.919 INFO:teuthology.run_tasks:Running task kernel... 2024-05-15T12:22:41.005 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-15T12:22:41.005 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-15T12:22:41.006 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:22:41.006 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-15T12:22:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:22:41.022 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-05-15T12:22:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:22:41.023 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-05-15T12:22:41.078 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-437.el9.x86_64 2024-05-15T12:22:41.078 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.14.0-437.el9.x86_64 2024-05-15T12:22:41.078 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2024-05-15T12:22:41.081 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-437.el9.x86_64 2024-05-15T12:22:41.081 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-437.el9.x86_64 2024-05-15T12:22:41.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-05-15T12:22:43.548 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.1 MB 00:00 2024-05-15T12:22:44.156 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.1 MB 00:01 2024-05-15T12:22:45.232 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-05-15T12:22:45.957 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-05-15T12:22:51.133 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.2 MB 00:01 2024-05-15T12:22:52.049 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 1.9 MB/s | 6.2 MB 00:03 2024-05-15T12:22:52.858 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-05-15T12:22:53.608 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 16 kB 00:00 2024-05-15T12:22:54.001 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-05-15T12:22:56.271 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-05-15T12:22:59.185 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-05-15T12:23:01.174 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-15T12:23:01.175 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-15T12:23:01.269 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-05-15T12:23:01.276 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-05-15T12:23:01.276 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-05-15T12:23:01.276 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-05-15T12:23:01.276 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-05-15T12:23:01.276 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-05-15T12:23:01.277 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:23:01.278 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 78 M 2024-05-15T12:23:01.278 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 123 M 2024-05-15T12:23:01.278 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-05-15T12:23:01.409 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-15T12:23:01.711 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 1.9 MB/s | 550 kB 00:00 2024-05-15T12:23:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-15T12:23:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-15T12:23:03.452 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 7.6 MB/s | 15 MB 00:02 2024-05-15T12:23:03.511 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-05-15T12:23:03.516 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-05-15T12:23:03.516 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout: kernel x86_64 5.14.0-446.el9 baseos 550 k 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core x86_64 5.14.0-446.el9 baseos 15 M 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules x86_64 5.14.0-446.el9 baseos 34 M 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core x86_64 5.14.0-446.el9 baseos 28 M 2024-05-15T12:23:03.517 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:23:03.518 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-05-15T12:23:03.518 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-05-15T12:23:03.518 INFO:teuthology.orchestra.run.smithi102.stdout:Install 4 Packages 2024-05-15T12:23:03.518 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:23:03.519 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 78 M 2024-05-15T12:23:03.519 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 123 M 2024-05-15T12:23:03.519 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-05-15T12:23:03.769 INFO:teuthology.orchestra.run.smithi102.stdout:(1/4): kernel-5.14.0-446.el9.x86_64.rpm 2.5 MB/s | 550 kB 00:00 2024-05-15T12:23:05.153 INFO:teuthology.orchestra.run.smithi102.stdout:(2/4): kernel-core-5.14.0-446.el9.x86_64.rpm 9.7 MB/s | 15 MB 00:01 2024-05-15T12:23:05.519 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 8.3 MB/s | 34 MB 00:04 2024-05-15T12:23:05.795 INFO:teuthology.orchestra.run.smithi102.stdout:(3/4): kernel-modules-5.14.0-446.el9.x86_64.rpm 15 MB/s | 34 MB 00:02 2024-05-15T12:23:06.053 INFO:teuthology.orchestra.run.smithi102.stdout:(4/4): kernel-modules-core-5.14.0-446.el9.x86_6 12 MB/s | 28 MB 00:02 2024-05-15T12:23:06.054 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:23:06.054 INFO:teuthology.orchestra.run.smithi102.stdout:Total 31 MB/s | 78 MB 00:02 2024-05-15T12:23:06.734 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-05-15T12:23:07.074 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-05-15T12:23:07.074 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-05-15T12:23:08.006 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-05-15T12:23:08.006 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-05-15T12:23:08.794 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-core-5.14.0-446.el9.x86_6 4.0 MB/s | 28 MB 00:07 2024-05-15T12:23:08.794 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:23:08.794 INFO:teuthology.orchestra.run.smithi173.stdout:Total 10 MB/s | 78 MB 00:07 2024-05-15T12:23:09.481 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-05-15T12:23:09.823 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-05-15T12:23:09.823 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-05-15T12:23:10.433 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-05-15T12:23:10.746 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-05-15T12:23:10.747 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-05-15T12:23:11.052 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:23:11.137 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:13.133 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-05-15T12:23:13.806 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:13.836 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:23:13.919 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:13.942 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:16.585 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:16.737 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:18.692 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:18.746 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:21.432 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:21.540 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:23.494 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:26.277 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:52.805 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:52.843 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:53.115 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:53.115 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:23:53.115 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:53.115 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:23:53.942 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-05-15T12:23:54.282 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:23:54.358 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:23:54.359 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:23:54.359 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-15T12:23:54.359 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:23:54.359 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:23:54.359 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-15T12:23:54.359 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:23:54.359 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:23:54.409 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:23:54.409 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:23:54.409 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-05-15T12:23:54.449 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-15T12:23:54.450 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:23:54.451 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:23:54.452 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:23:54.452 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 5.14.0-446.el9.x86_64 2024-05-15T12:23:54.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:23:55.051 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-05-15T12:23:55.714 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:55.753 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:56.048 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:56.048 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-446.el9.x86_64 1/4 2024-05-15T12:23:56.048 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-446.el9.x86_64 2/4 2024-05-15T12:23:56.048 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-446.el9.x86_64 3/4 2024-05-15T12:23:56.527 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-15T12:23:56.552 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-05-15T12:23:56.554 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-05-15T12:23:56.570 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.RWN1NMMQFw 2024-05-15T12:23:56.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RWN1NMMQFw 2024-05-15T12:23:56.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.RWN1NMMQFw 2024-05-15T12:23:56.783 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.RWN1NMMQFw is 6KB 2024-05-15T12:23:56.832 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.RWN1NMMQFw 2024-05-15T12:23:56.846 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:23:56.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-446.el9.x86_64 4/4 2024-05-15T12:23:56.847 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:23:56.847 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-05-15T12:23:56.847 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-446.el9.x86_64 2024-05-15T12:23:56.847 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-446.el9.x86_64 2024-05-15T12:23:56.848 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-446.el9.x86_64 2024-05-15T12:23:56.848 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-446.el9.x86_64 2024-05-15T12:23:56.848 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:23:56.848 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-05-15T12:23:56.907 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-15T12:23:56.907 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-15T12:23:56.907 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-15T12:23:56.907 INFO:teuthology.orchestra.run.smithi102.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-15T12:23:56.909 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-15T12:23:57.075 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-05-15T12:23:57.213 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:23:57.301 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:23:57.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:23:57.302 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-446.el9.x86_64 2024-05-15T12:23:57.302 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-15T12:23:57.302 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-15T12:23:57.302 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-15T12:23:57.302 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-05-15T12:23:57.302 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-05-15T12:23:57.352 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-446.el9.x86_64 2024-05-15T12:23:57.352 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-446.el9.x86_64 2024-05-15T12:23:57.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-05-15T12:23:57.383 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-15T12:23:57.384 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:23:57.385 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-05-15T12:23:57.386 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-15T12:23:57.387 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-446.el9.x86_64 2024-05-15T12:23:57.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-15T12:23:57.975 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-05-15T12:23:59.349 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-15T12:23:59.373 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-05-15T12:23:59.374 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-05-15T12:23:59.390 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.qyYw4Mn5uX 2024-05-15T12:23:59.390 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.qyYw4Mn5uX 2024-05-15T12:23:59.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.qyYw4Mn5uX 2024-05-15T12:23:59.594 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.qyYw4Mn5uX is 6KB 2024-05-15T12:23:59.643 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.qyYw4Mn5uX 2024-05-15T12:23:59.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-05-15T12:23:59.726 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-15T12:23:59.726 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-15T12:23:59.727 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-15T12:23:59.727 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64.conf 2024-05-15T12:23:59.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-446.el9.x86_64 2024-05-15T12:23:59.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-05-15T12:24:27.104 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-15T12:24:27.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-05-15T12:24:27.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:24:29.881 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-15T12:24:29.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-15T12:24:29.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:24:45.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-05-15T12:24:48.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-05-15T12:24:54.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-05-15T12:24:54.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:24:57.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-15T12:24:57.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:25:00.482 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-05-15T12:25:12.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-15T12:25:12.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:25:44.233 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-05-15T12:25:44.764 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-15T12:25:44.765 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-446.el9.x86_64"... 2024-05-15T12:25:44.765 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-05-15T12:25:44.780 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-446.el9.x86_64 2024-05-15T12:25:44.780 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-15T12:25:44.780 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:25:44.780 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:25:45.781 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-15T12:25:45.781 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:25:45.844 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-05-15T12:25:45.875 DEBUG:teuthology.parallel:result is None 2024-05-15T12:25:54.606 DEBUG:teuthology.orchestra.remote:timed out 2024-05-15T12:26:06.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-05-15T12:26:06.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:26:06.890 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-05-15T12:26:07.251 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-05-15T12:26:07.251 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-446.el9.x86_64"... 2024-05-15T12:26:07.251 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-05-15T12:26:07.268 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-446.el9.x86_64 2024-05-15T12:26:07.268 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-446.el9.x86_64 vs 5.14.0-446.el9.x86_64 2024-05-15T12:26:07.268 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-15T12:26:07.268 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-15T12:26:08.269 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-15T12:26:08.269 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-15T12:26:08.347 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-05-15T12:26:08.387 DEBUG:teuthology.parallel:result is None 2024-05-15T12:26:08.387 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-15T12:26:08.394 INFO:teuthology.task.internal:Creating test directory... 2024-05-15T12:26:08.394 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:26:08.397 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-15T12:26:08.416 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-15T12:26:08.436 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-15T12:26:08.494 INFO:teuthology.task.internal:Creating archive directory... 2024-05-15T12:26:08.495 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:26:08.497 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-15T12:26:08.617 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-15T12:26:08.624 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-15T12:26:08.625 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:26:08.627 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-15T12:26:08.661 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.661 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.672 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.672 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-15T12:26:08.675 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-15T12:26:08.682 INFO:teuthology.task.internal:Configuring sudo... 2024-05-15T12:26:08.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:26:08.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-15T12:26:08.746 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-15T12:26:08.760 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-15T12:26:08.761 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:26:08.786 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-15T12:26:08.804 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:08.888 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:08.931 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:09.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:09.058 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:26:09.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:26:09.120 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:09.166 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-15T12:26:09.220 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:09.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-15T12:26:09.350 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:26:09.350 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-15T12:26:09.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-05-15T12:26:09.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-05-15T12:26:09.449 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:26:09.507 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:26:09.942 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-15T12:26:09.950 INFO:teuthology.task.internal:Starting timer... 2024-05-15T12:26:09.950 INFO:teuthology.run_tasks:Running task pcp... 2024-05-15T12:26:09.980 INFO:teuthology.run_tasks:Running task selinux... 2024-05-15T12:26:10.028 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-05-15T12:26:10.029 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-05-15T12:26:10.085 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-05-15T12:26:10.087 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-05-15T12:26:10.140 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-05-15T12:26:10.141 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-15T12:26:10.142 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-05-15T12:26:10.168 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-05-15T12:26:10.168 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-05-15T12:26:10.190 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-05-15T12:26:10.190 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-05-15T12:26:10.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:26:10.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:26:10.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:26:10.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:26:10.264 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-15T12:26:10.264 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:26:10.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-05-15T12:26:10.331 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-15T12:26:10.340 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-15T12:26:10.360 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-15T12:26:10.361 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-15T12:30:26.629 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-05-15T12:30:26.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-05-15T12:30:26.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:30:26.708 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-05-15T12:30:26.791 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-05-15T12:30:26.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-15T12:30:26.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-15T12:30:26.867 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-05-15T12:30:26.951 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-05-15T12:30:26.951 INFO:teuthology.run_tasks:Running task clock... 2024-05-15T12:30:26.961 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-15T12:30:26.962 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:30:26.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:30:26.964 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-15T12:30:26.964 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:30:26.993 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:30:27.008 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:30:27.021 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-15T12:30:27.033 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-05-15T12:30:27.035 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-15T12:30:27.044 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-05-15T12:30:27.059 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:30:27.060 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-05-15T12:30:27.071 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-05-15T12:30:27.073 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:30:27.085 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-15T12:30:27.098 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-15T12:30:27.127 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:30:27.130 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:30:27.130 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-05-15T12:30:27.131 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.131 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.131 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.131 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.154 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:30:27.158 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:30:27.159 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-05-15T12:30:27.159 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.159 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.159 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.159 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:30:27.159 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-05-15T12:30:27.170 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-05-15T12:30:27.170 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:30:27.170 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:30:27.188 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:30:27.188 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-05-15T12:30:27.244 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:30:27.244 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:27.244 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-15T12:30:27.244 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:27.244 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:27.245 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:30:25.669914950 +0000 2024-05-15T12:30:27.245 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:29:47.501665157 +0000 2024-05-15T12:30:27.245 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:29:47.501665157 +0000 2024-05-15T12:30:27.245 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:29:47.501665157 +0000 2024-05-15T12:30:27.245 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:30:27.308 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:30:27.308 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:30:27.308 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000250281 s, 2.0 MB/s 2024-05-15T12:30:27.309 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:30:27.366 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:30:25.669914950 +0000 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:29:47.860658030 +0000 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:29:47.860658030 +0000 2024-05-15T12:30:27.422 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:29:47.860658030 +0000 2024-05-15T12:30:27.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:30:27.486 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:30:27.486 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:30:27.486 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000184292 s, 2.8 MB/s 2024-05-15T12:30:27.488 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:30:27.544 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-05-15T12:30:27.599 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:30:27.599 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:27.600 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-15T12:30:27.600 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:27.600 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:27.600 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:30:25.669914950 +0000 2024-05-15T12:30:27.600 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:29:48.204651202 +0000 2024-05-15T12:30:27.600 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:29:48.204651202 +0000 2024-05-15T12:30:27.600 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:29:48.204651202 +0000 2024-05-15T12:30:27.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:30:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:30:27.662 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:30:27.663 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000165638 s, 3.1 MB/s 2024-05-15T12:30:27.663 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:30:27.719 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-05-15T12:30:27.775 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:30:27.775 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:27.775 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-05-15T12:30:27.775 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:27.775 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:30:25.669914950 +0000 2024-05-15T12:30:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:29:48.539644552 +0000 2024-05-15T12:30:27.777 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:29:48.539644552 +0000 2024-05-15T12:30:27.777 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:29:48.539644552 +0000 2024-05-15T12:30:27.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:30:27.838 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:30:27.838 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:30:27.838 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000221899 s, 2.3 MB/s 2024-05-15T12:30:27.839 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:30:27.895 DEBUG:teuthology.orchestra.run.smithi102:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-05-15T12:30:28.035 INFO:teuthology.orchestra.run.smithi102.stdout:loop 2024-05-15T12:30:28.036 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_1... 2024-05-15T12:30:28.036 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-05-15T12:30:28.068 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-05-15T12:30:28.097 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_11 2024-05-15T12:30:28.121 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_2... 2024-05-15T12:30:28.122 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-05-15T12:30:28.194 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-05-15T12:30:28.223 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_21 2024-05-15T12:30:28.247 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_3... 2024-05-15T12:30:28.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-05-15T12:30:28.320 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-05-15T12:30:28.349 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_31 2024-05-15T12:30:28.384 INFO:tasks.nvme_loop:Connecting nvme_loop smithi102:/dev/vg_nvme/lv_4... 2024-05-15T12:30:28.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-05-15T12:30:28.454 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-05-15T12:30:28.484 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/vg_nvme/lv_41 2024-05-15T12:30:28.509 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:30:28.509 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:30:28.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme list 2024-05-15T12:30:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:Node SN Model Namespace Usage Format FW Rev 2024-05-15T12:30:28.628 INFO:teuthology.orchestra.run.smithi102.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-05-15T12:30:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme0n1 PHFT620400VK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-05-15T12:30:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme1n1 a873206239e436837536 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme2n1 a25c89325b5ec4b0cb0d Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme3n1 113ad7a434a903ab7acf Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:28.629 INFO:teuthology.orchestra.run.smithi102.stdout:/dev/nvme4n1 fc045c8e8a02760fd1ef Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:28.630 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-05-15T12:30:28.630 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:30:28.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/scratch_devs 2024-05-15T12:30:28.695 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:30:28.695 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:30:28.715 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-15T12:30:28.715 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-05-15T12:30:28.773 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-15T12:30:28.773 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:28.773 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-05-15T12:30:28.774 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:28.774 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:28.774 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:30:25.643623086 +0000 2024-05-15T12:30:28.774 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:29:47.514443264 +0000 2024-05-15T12:30:28.774 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:29:47.514443264 +0000 2024-05-15T12:30:28.774 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:29:47.514443264 +0000 2024-05-15T12:30:28.774 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-15T12:30:28.842 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:30:28.842 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:30:28.842 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000278353 s, 1.8 MB/s 2024-05-15T12:30:28.843 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-15T12:30:28.902 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-05-15T12:30:28.959 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:30:25.643623086 +0000 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:29:47.879435410 +0000 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:29:47.879435410 +0000 2024-05-15T12:30:28.960 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:29:47.879435410 +0000 2024-05-15T12:30:28.960 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-15T12:30:29.025 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:30:29.025 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:30:29.025 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00023124 s, 2.2 MB/s 2024-05-15T12:30:29.026 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-15T12:30:29.083 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:30:25.643623086 +0000 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:29:48.212428246 +0000 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:29:48.212428246 +0000 2024-05-15T12:30:29.138 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:29:48.212428246 +0000 2024-05-15T12:30:29.139 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-15T12:30:29.201 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:30:29.201 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:30:29.201 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000274041 s, 1.9 MB/s 2024-05-15T12:30:29.202 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-15T12:30:29.259 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-05-15T12:30:29.314 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-15T12:30:29.314 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-15T12:30:29.314 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-15T12:30:29.314 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-15T12:30:29.314 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-05-15T12:30:29.314 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:30:25.643623086 +0000 2024-05-15T12:30:29.314 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:29:48.543421125 +0000 2024-05-15T12:30:29.315 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:29:48.543421125 +0000 2024-05-15T12:30:29.315 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:29:48.543421125 +0000 2024-05-15T12:30:29.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-15T12:30:29.377 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:30:29.378 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:30:29.378 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000179908 s, 2.8 MB/s 2024-05-15T12:30:29.379 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-15T12:30:29.435 DEBUG:teuthology.orchestra.run.smithi173:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-05-15T12:30:29.605 INFO:teuthology.orchestra.run.smithi173.stdout:loop 2024-05-15T12:30:29.606 INFO:tasks.nvme_loop:Connecting nvme_loop smithi173:/dev/vg_nvme/lv_1... 2024-05-15T12:30:29.606 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-05-15T12:30:29.639 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-05-15T12:30:29.668 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/vg_nvme/lv_11 2024-05-15T12:30:29.692 INFO:tasks.nvme_loop:Connecting nvme_loop smithi173:/dev/vg_nvme/lv_2... 2024-05-15T12:30:29.692 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-05-15T12:30:29.772 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-05-15T12:30:29.801 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/vg_nvme/lv_21 2024-05-15T12:30:29.835 INFO:tasks.nvme_loop:Connecting nvme_loop smithi173:/dev/vg_nvme/lv_3... 2024-05-15T12:30:29.835 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-05-15T12:30:29.907 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-05-15T12:30:29.936 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/vg_nvme/lv_31 2024-05-15T12:30:29.959 INFO:tasks.nvme_loop:Connecting nvme_loop smithi173:/dev/vg_nvme/lv_4... 2024-05-15T12:30:29.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-05-15T12:30:30.034 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-05-15T12:30:30.063 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/vg_nvme/lv_41 2024-05-15T12:30:30.093 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:30:30.094 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:30:30.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo nvme list 2024-05-15T12:30:30.213 INFO:teuthology.orchestra.run.smithi173.stdout:Node SN Model Namespace Usage Format FW Rev 2024-05-15T12:30:30.213 INFO:teuthology.orchestra.run.smithi173.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-05-15T12:30:30.213 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/nvme0n1 PHFT6204019S400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-05-15T12:30:30.213 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/nvme1n1 a8deeb0d7a44f3b7496c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:30.214 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/nvme2n1 606caf7cf3cc80abb52c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:30.214 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/nvme3n1 006111c4528f6f135821 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:30.214 INFO:teuthology.orchestra.run.smithi173.stdout:/dev/nvme4n1 db7ff066689ac85b4735 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-05-15T12:30:30.215 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-05-15T12:30:30.215 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:30:30.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/scratch_devs 2024-05-15T12:30:30.280 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-15T12:30:30.379 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN'], 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314'} 2024-05-15T12:30:30.379 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:30:30.380 INFO:tasks.cephadm:Cluster fsid is eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:30:30.380 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-15T12:30:30.380 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.102', 'mon.c': '[v2:172.21.15.102:3301,v1:172.21.15.102:6790]', 'mon.b': '172.21.15.173'} 2024-05-15T12:30:30.380 INFO:tasks.cephadm:First mon is mon.a on smithi102 2024-05-15T12:30:30.380 INFO:tasks.cephadm:First mgr is y 2024-05-15T12:30:30.380 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-15T12:30:30.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-05-15T12:30:30.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-05-15T12:30:30.435 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-15T12:30:30.438 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:30:30.668 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'ae2481253f9b2fa3451954e615eedb67ebdd5314', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79382/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1882-gae248125', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1882.gae248125'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-14 20:03:06.787269', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-osd-ec-partial-reads', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-15T12:30:30.883 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-osd-ec-partial-reads, sha1 ae2481253f9b2fa3451954e615eedb67ebdd5314 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:30:30.884 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/x86_64/flavors/default/cephadm 2024-05-15T12:30:30.884 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T12:30:32.149 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r--. 1 ubuntu ubuntu 767678 May 15 12:30 /home/ubuntu/cephtest/cephadm 2024-05-15T12:30:32.149 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-osd-ec-partial-reads/ae2481253f9b2fa3451954e615eedb67ebdd5314/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-15T12:30:33.024 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 767678 May 15 12:30 /home/ubuntu/cephtest/cephadm 2024-05-15T12:30:33.024 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T12:30:33.043 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-15T12:30:33.073 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 on all hosts... 2024-05-15T12:30:33.073 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 pull 2024-05-15T12:30:33.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 pull 2024-05-15T12:30:33.856 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:30:34.037 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:31:06.980 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-05-15T12:31:06.980 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev)", 2024-05-15T12:31:06.980 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9", 2024-05-15T12:31:06.980 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-05-15T12:31:06.980 INFO:teuthology.orchestra.run.smithi102.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328" 2024-05-15T12:31:06.980 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-05-15T12:31:06.981 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-05-15T12:31:11.651 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-05-15T12:31:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev)", 2024-05-15T12:31:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9", 2024-05-15T12:31:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-05-15T12:31:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328" 2024-05-15T12:31:11.651 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-05-15T12:31:11.651 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-05-15T12:31:11.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-05-15T12:31:11.708 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-05-15T12:31:11.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-05-15T12:31:11.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-05-15T12:31:11.800 INFO:tasks.cephadm:Writing seed config... 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-15T12:31:11.802 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-15T12:31:11.803 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-15T12:31:11.803 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-15T12:31:11.803 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:31:11.803 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-15T12:31:11.832 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eac36fa0-12b6-11ef-bc98-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-05-15T12:31:11.833 DEBUG:teuthology.orchestra.run.smithi102:mon.a> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a.service 2024-05-15T12:31:11.876 DEBUG:teuthology.orchestra.run.smithi102:mgr.y> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.y.service 2024-05-15T12:31:11.919 INFO:tasks.cephadm:Bootstrapping... 2024-05-15T12:31:11.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 -v bootstrap --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.102 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:31:12.112 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-05-15T12:31:12.112 INFO:teuthology.orchestra.run.smithi102.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314', '-v', 'bootstrap', '--fsid', 'eac36fa0-12b6-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.102', '--skip-admin-label'] 2024-05-15T12:31:12.137 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 5.0.2 2024-05-15T12:31:12.138 INFO:teuthology.orchestra.run.smithi102.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-15T12:31:12.138 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying podman|docker is present... 2024-05-15T12:31:12.164 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 5.0.2 2024-05-15T12:31:12.164 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying lvm2 is present... 2024-05-15T12:31:12.164 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying time synchronization is in place... 2024-05-15T12:31:12.171 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-15T12:31:12.171 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-15T12:31:12.177 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-15T12:31:12.178 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-05-15T12:31:12.184 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-05-15T12:31:12.190 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-05-15T12:31:12.190 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-05-15T12:31:12.190 INFO:teuthology.orchestra.run.smithi102.stdout:Repeating the final host check... 2024-05-15T12:31:12.214 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout 5.0.2 2024-05-15T12:31:12.214 INFO:teuthology.orchestra.run.smithi102.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-15T12:31:12.214 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl is present 2024-05-15T12:31:12.214 INFO:teuthology.orchestra.run.smithi102.stdout:lvcreate is present 2024-05-15T12:31:12.220 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-15T12:31:12.220 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-15T12:31:12.226 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-15T12:31:12.226 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-05-15T12:31:12.233 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout enabled 2024-05-15T12:31:12.239 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout active 2024-05-15T12:31:12.239 INFO:teuthology.orchestra.run.smithi102.stdout:Unit chronyd.service is enabled and running 2024-05-15T12:31:12.239 INFO:teuthology.orchestra.run.smithi102.stdout:Host looks OK 2024-05-15T12:31:12.239 INFO:teuthology.orchestra.run.smithi102.stdout:Cluster fsid: eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:12.239 INFO:teuthology.orchestra.run.smithi102.stdout:Acquiring lock 139920541343312 on /run/cephadm/eac36fa0-12b6-11ef-bc98-c7b262605968.lock 2024-05-15T12:31:12.239 INFO:teuthology.orchestra.run.smithi102.stdout:Lock 139920541343312 acquired on /run/cephadm/eac36fa0-12b6-11ef-bc98-c7b262605968.lock 2024-05-15T12:31:12.239 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 3300 ... 2024-05-15T12:31:12.240 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying IP 172.21.15.102 port 6789 ... 2024-05-15T12:31:12.240 INFO:teuthology.orchestra.run.smithi102.stdout:Base mon IP(s) is [172.21.15.102:3300, 172.21.15.102:6789], mon addrv is [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-05-15T12:31:12.243 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.102 metric 100 2024-05-15T12:31:12.243 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.102 metric 100 2024-05-15T12:31:12.246 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-15T12:31:12.246 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-15T12:31:12.246 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2024-05-15T12:31:12.248 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-15T12:31:12.248 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-15T12:31:12.248 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-15T12:31:12.248 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-15T12:31:12.248 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-05-15T12:31:12.248 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-15T12:31:12.249 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-05-15T12:31:12.249 INFO:teuthology.orchestra.run.smithi102.stdout:Mon IP `172.21.15.102` is in CIDR network `172.21.0.0/20` 2024-05-15T12:31:12.249 INFO:teuthology.orchestra.run.smithi102.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-15T12:31:12.249 INFO:teuthology.orchestra.run.smithi102.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-15T12:31:12.250 INFO:teuthology.orchestra.run.smithi102.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:31:12.774 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stdout bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 2024-05-15T12:31:12.774 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314... 2024-05-15T12:31:12.774 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Getting image source signatures 2024-05-15T12:31:12.774 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:4502bcc40a1bb6c3aa83c1a9c4cc55deee56ecb239477549ee4bf66585c31c94 2024-05-15T12:31:12.774 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying blob sha256:d0e808cd455827ca3108db1b0125dcee9ffc148daa1d7dde963f550bc831d501 2024-05-15T12:31:12.774 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Copying config sha256:bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 2024-05-15T12:31:12.774 INFO:teuthology.orchestra.run.smithi102.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-15T12:31:13.678 INFO:teuthology.orchestra.run.smithi102.stdout:ceph: stdout ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev) 2024-05-15T12:31:13.678 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph version: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev) 2024-05-15T12:31:13.679 INFO:teuthology.orchestra.run.smithi102.stdout:Extracting ceph user uid/gid from container image... 2024-05-15T12:31:14.685 INFO:teuthology.orchestra.run.smithi102.stdout:stat: stdout 167 167 2024-05-15T12:31:14.686 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial keys... 2024-05-15T12:31:15.608 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQATq0RmC/FqAxAAmHhhUVDPE2RULY0uWcor4w== 2024-05-15T12:31:16.505 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQATq0RmSx5bNhAA+8wQDIlQqysfrtBK8hjLHg== 2024-05-15T12:31:17.435 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph-authtool: stdout AQAUq0Rmvb1oMxAAUwtinzMGKjR7EgI46Qh1kg== 2024-05-15T12:31:17.436 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial monmap... 2024-05-15T12:31:18.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-15T12:31:18.409 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool for a [v2:172.21.15.102:3300,v1:172.21.15.102:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = quincy 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: set fsid to eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:31:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mon... 2024-05-15T12:31:19.356 INFO:teuthology.orchestra.run.smithi102.stdout:create mon.a on 2024-05-15T12:31:19.724 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-15T12:31:19.893 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968.target. 2024-05-15T12:31:19.893 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968.target. 2024-05-15T12:31:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a 2024-05-15T12:31:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-15T12:31:20.307 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968.target.wants/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@.service. 2024-05-15T12:31:20.864 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:31:20.865 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-05-15T12:31:20.872 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:31:20.872 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-05-15T12:31:20.872 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-05-15T12:31:20.872 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-05-15T12:31:20.872 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon to start... 2024-05-15T12:31:20.872 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mon... 2024-05-15T12:31:20.896 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:20 smithi102 systemd[1]: Started Ceph mon.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:31:21.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:21 smithi102 ceph-mon[24426]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-15T12:31:21.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:21 smithi102 ceph-mon[24426]: monmap e1: 1 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:31:21.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:21 smithi102 ceph-mon[24426]: fsmap 2024-05-15T12:31:21.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:21 smithi102 ceph-mon[24426]: osdmap e1: 0 total, 0 up, 0 in 2024-05-15T12:31:21.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:21 smithi102 ceph-mon[24426]: mgrmap e1: no daemons active 2024-05-15T12:31:21.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:21 smithi102 ceph-mon[24426]: from='client.? 172.21.15.102:0/683961447' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-15T12:31:21.910 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout cluster: 2024-05-15T12:31:21.910 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout id: eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout services: 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.539738s) 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout data: 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout pgs: 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:mon is available 2024-05-15T12:31:21.911 INFO:teuthology.orchestra.run.smithi102.stdout:Assimilating anything we can from ceph.conf... 2024-05-15T12:31:23.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:22 smithi102 ceph-mon[24426]: from='client.? 172.21.15.102:0/3969382342' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-15T12:31:23.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:22 smithi102 ceph-mon[24426]: from='client.? 172.21.15.102:0/3969382342' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-15T12:31:23.016 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:23.016 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-05-15T12:31:23.016 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:23.016 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.102:3300,v1:172.21.15.102:6789] 2024-05-15T12:31:23.016 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-15T12:31:23.016 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-15T12:31:23.016 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-15T12:31:23.017 INFO:teuthology.orchestra.run.smithi102.stdout:Generating new minimal ceph.conf... 2024-05-15T12:31:24.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:23 smithi102 ceph-mon[24426]: from='client.? 172.21.15.102:0/789845650' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:31:24.041 INFO:teuthology.orchestra.run.smithi102.stdout:Restarting the monitor... 2024-05-15T12:31:24.314 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:24 smithi102 systemd[1]: Stopping Ceph mon.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:31:24.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:24 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a[24421]: 2024-05-15T12:31:24.324+0000 7fc7284d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:31:24.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:24 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a[24421]: 2024-05-15T12:31:24.324+0000 7fc7284d5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-15T12:31:24.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:24 smithi102 podman[24674]: 2024-05-15 12:31:24.51859803 +0000 UTC m=+0.302911718 container died 01b45e29468d2d2767608008377c8762f5114727311df5032c135ef85396c528 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-15T12:31:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:24 smithi102 podman[24674]: 2024-05-15 12:31:24.832474035 +0000 UTC m=+0.616787722 container cleanup 01b45e29468d2d2767608008377c8762f5114727311df5032c135ef85396c528 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, ceph=True, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-15T12:31:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:24 smithi102 bash[24674]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a 2024-05-15T12:31:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 podman[24687]: 2024-05-15 12:31:25.016774842 +0000 UTC m=+0.501914864 container remove 01b45e29468d2d2767608008377c8762f5114727311df5032c135ef85396c528 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-osd-ec-partial-reads-ae24812, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514) 2024-05-15T12:31:25.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-15T12:31:25.598 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 systemd[1]: Stopped Ceph mon.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:31:25.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 systemd[1]: Starting Ceph mon.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:31:25.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 podman[24780]: 2024-05-15 12:31:25.362611709 +0000 UTC m=+0.016953237 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:31:25.599 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 podman[24780]: 2024-05-15 12:31:25.467495684 +0000 UTC m=+0.121837193 container create 6969c783f08fb4e622124f5fc247feeafcf37b1ef49f3b9347ad68775d785ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-15T12:31:25.821 INFO:teuthology.orchestra.run.smithi102.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-15T12:31:26.032 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 podman[24780]: 2024-05-15 12:31:25.753528965 +0000 UTC m=+0.407870486 container init 6969c783f08fb4e622124f5fc247feeafcf37b1ef49f3b9347ad68775d785ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 podman[24780]: 2024-05-15 12:31:25.75686944 +0000 UTC m=+0.411210960 container start 6969c783f08fb4e622124f5fc247feeafcf37b1ef49f3b9347ad68775d785ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240514, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: set uid:gid to 167:167 (ceph:ceph) 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev), process ceph-mon, pid 2 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: pidfile_write: ignore empty --pid-file 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: load: jerasure load: lrc 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: RocksDB version: 7.9.2 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Git sha 0 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Compile date 2024-05-14 19:11:07 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: DB SUMMARY 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: DB Session ID: O3MHABXRTEJT7VVM5J0Z 2024-05-15T12:31:26.033 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: CURRENT file: CURRENT 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: IDENTITY file: IDENTITY 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84647 ; 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.error_if_exists: 0 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.create_if_missing: 0 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.paranoid_checks: 1 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.env: 0x562ad9ddeb20 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.fs: PosixFileSystem 2024-05-15T12:31:26.034 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.info_log: 0x562adb2f3900 2024-05-15T12:31:26.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_file_opening_threads: 16 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.statistics: (nil) 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.use_fsync: 0 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_log_file_size: 0 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.keep_log_file_num: 1000 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.recycle_log_file_num: 0 2024-05-15T12:31:26.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.allow_fallocate: 1 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.allow_mmap_reads: 0 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.allow_mmap_writes: 0 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.use_direct_reads: 0 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.create_missing_column_families: 0 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.db_log_dir: 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.wal_dir: 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.advise_random_on_open: 1 2024-05-15T12:31:26.037 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.db_write_buffer_size: 0 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.write_buffer_manager: 0x562adb2c7400 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.rate_limiter: (nil) 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.wal_recovery_mode: 2 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.enable_thread_tracking: 0 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.enable_pipelined_write: 0 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.unordered_write: 0 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-15T12:31:26.038 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.row_cache: None 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.wal_filter: None 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.allow_ingest_behind: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.two_write_queues: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.manual_wal_flush: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.wal_compression: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.atomic_flush: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.log_readahead_size: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.best_efforts_recovery: 0 2024-05-15T12:31:26.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.allow_data_in_errors: 0 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.db_host_id: __hostname__ 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_background_jobs: 2 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_background_compactions: -1 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_subcompactions: 1 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_total_wal_size: 0 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-15T12:31:26.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_open_files: -1 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bytes_per_sync: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_readahead_size: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_background_flushes: -1 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Compression algorithms supported: 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kZSTD supported: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kXpressCompression supported: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kLZ4HCCompression supported: 1 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kZlibCompression supported: 1 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kSnappyCompression supported: 1 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kLZ4Compression supported: 1 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: kBZip2Compression supported: 0 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-15T12:31:26.041 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.merge_operator: 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_filter: None 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_filter_factory: None 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.sst_partitioner_factory: None 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562adb2f3a00) 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-15T12:31:26.042 INFO:journalctl@ceph.mon.a.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_type: 0 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_index_type: 0 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_shortening: 1 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: checksum: 4 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: no_block_cache: 0 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache: 0x562adb2f9090 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_options: 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: capacity : 536870912 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_shard_bits : 4 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: strict_capacity_limit : 0 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_cache_compressed: (nil) 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: persistent_cache: (nil) 2024-05-15T12:31:26.043 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size: 4096 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_size_deviation: 10 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_restart_interval: 16 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: index_block_restart_interval: 1 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: metadata_block_size: 4096 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: partition_filters: 0 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: use_delta_encoding: 1 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: filter_policy: bloomfilter 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: whole_key_filtering: 1 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: verify_compression: 0 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: format_version: 5 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_index_compression: 1 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: block_align: 0 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: max_auto_readahead_size: 262144 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: prepopulate_block_cache: 0 2024-05-15T12:31:26.044 INFO:journalctl@ceph.mon.a.smithi102.stdout: initial_auto_readahead_size: 8192 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.write_buffer_size: 33554432 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_write_buffer_number: 2 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression: NoCompression 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression: Disabled 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.prefix_extractor: nullptr 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.num_levels: 7 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-15T12:31:26.045 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.level: 32767 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.strategy: 0 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.enabled: false 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-15T12:31:26.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.target_file_size_base: 67108864 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-15T12:31:26.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.arena_block_size: 1048576 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.disable_auto_compactions: 0 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-15T12:31:26.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.table_properties_collectors: 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.inplace_update_support: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.bloom_locality: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.max_successive_merges: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.paranoid_file_checks: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.force_consistency_checks: 1 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.report_bg_io_stats: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.ttl: 2592000 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.enable_blob_files: false 2024-05-15T12:31:26.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.min_blob_size: 0 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.blob_file_size: 268435456 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.blob_file_starting_level: 0 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8a6541c7-0e57-4451-bc97-3f6b72aeacd3 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776285785959, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776285786875, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81272, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79459, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73768, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715776285, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8a6541c7-0e57-4451-bc97-3f6b72aeacd3", "db_session_id": "O3MHABXRTEJT7VVM5J0Z", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-15T12:31:26.050 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776285786961, "job": 1, "event": "recovery_finished"} 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562adb3ec000 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: DB pointer 0x562adb3d6000 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** DB Stats ** 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-05-15T12:31:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: L0 2/0 81.20 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: Sum 2/0 81.20 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 126.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** Compaction Stats [default] ** 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 126.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:31:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: Cumulative compaction: 0.00 GB write, 14.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: Interval compaction: 0.00 GB write, 14.02 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache BinnedLRUCache@0x562adb2f9090#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: starting mon.a rank 0 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???) e1 preinit fsid eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???).mds e1 new map 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???).mds e1 print_map 2024-05-15T12:31:26.053 INFO:journalctl@ceph.mon.a.smithi102.stdout: e1 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout: legacy client fscid: -1 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout: 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout: No filesystems configured 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 bash[24780]: 6969c783f08fb4e622124f5fc247feeafcf37b1ef49f3b9347ad68775d785ae4 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 systemd[1]: Started Ceph mon.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: monmap e1: 1 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:31:26.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: fsmap 2024-05-15T12:31:26.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: osdmap e1: 0 total, 0 up, 0 in 2024-05-15T12:31:26.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:25 smithi102 ceph-mon[24799]: mgrmap e1: no daemons active 2024-05-15T12:31:28.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:28 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/685696560' entity='client.admin' 2024-05-15T12:31:33.387 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-15T12:31:33.585 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:31:33.585 INFO:teuthology.orchestra.run.smithi102.stdout:Creating mgr... 2024-05-15T12:31:33.586 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-15T12:31:33.586 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-15T12:31:34.599 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.y 2024-05-15T12:31:34.599 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.y.service: Unit ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.y.service not loaded. 2024-05-15T12:31:34.827 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968.target.wants/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@.service. 2024-05-15T12:31:34.902 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:34 smithi102 systemd[1]: Starting Ceph mgr.y for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:31:35.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:35 smithi102 podman[25015]: 2024-05-15 12:31:34.97556141 +0000 UTC m=+0.017128226 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 2024-05-15T12:31:36.320 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:36 smithi102 podman[25015]: 2024-05-15 12:31:36.071228362 +0000 UTC m=+1.112795159 container create f859a519a0948da386644088f4cdbe3958c6798594eff6c87a1c8fe9eea712f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-15T12:31:37.248 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:31:37.248 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-05-15T12:31:37.256 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:31:37.256 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-05-15T12:31:37.256 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-05-15T12:31:37.256 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to enable service . firewalld.service is not available 2024-05-15T12:31:37.264 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:31:37.264 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-05-15T12:31:37.272 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:31:37.272 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-05-15T12:31:37.272 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-05-15T12:31:37.272 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-15T12:31:37.272 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr to start... 2024-05-15T12:31:37.272 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr... 2024-05-15T12:31:37.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 podman[25015]: 2024-05-15 12:31:37.160433281 +0000 UTC m=+2.202000084 container init f859a519a0948da386644088f4cdbe3958c6798594eff6c87a1c8fe9eea712f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-15T12:31:37.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 podman[25015]: 2024-05-15 12:31:37.163461837 +0000 UTC m=+2.205028641 container start f859a519a0948da386644088f4cdbe3958c6798594eff6c87a1c8fe9eea712f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-15T12:31:37.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 bash[25015]: f859a519a0948da386644088f4cdbe3958c6798594eff6c87a1c8fe9eea712f2 2024-05-15T12:31:37.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 systemd[1]: Started Ceph mgr.y for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:31:37.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:37.325+0000 7faa313d5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:31:37.694 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:37.516+0000 7faa313d5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:37.694 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:37.622+0000 7faa313d5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:31:38.018 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:37 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2741538069' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T12:31:38.019 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:37 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:37.944+0000 7faa313d5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:31:38.019 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:38 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:38.029+0000 7faa313d5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:31:38.352 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:38.352 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:31:38.352 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "eac36fa0-12b6-11ef-bc98-c7b262605968", 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-15T12:31:38.353 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T12:31:38.354 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T12:31:38.358 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:38.359 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T12:31:20.823853+0000", 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:31:38.360 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (1/15)... 2024-05-15T12:31:39.545 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:39 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:39.220+0000 7faa313d5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:31:39.545 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:39 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:39.387+0000 7faa313d5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:39.545 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:39 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:39.555+0000 7faa313d5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:31:40.119 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:39 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:39.839+0000 7faa313d5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:31:40.119 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:39 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:39.930+0000 7faa313d5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:31:40.119 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:40 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:40.129+0000 7faa313d5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:31:40.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:40 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:40.318+0000 7faa313d5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:41.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:40 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/951796263' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T12:31:41.152 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:40 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:40.945+0000 7faa313d5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:31:41.456 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:41.369+0000 7faa313d5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:31:41.456 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:41.466+0000 7faa313d5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:31:41.477 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "eac36fa0-12b6-11ef-bc98-c7b262605968", 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T12:31:41.478 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T12:31:41.479 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:41.480 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T12:31:20.823853+0000", 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T12:31:41.481 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:31:41.482 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (2/15)... 2024-05-15T12:31:41.902 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:41.667+0000 7faa313d5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:31:42.511 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:42 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:42.190+0000 7faa313d5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:31:42.511 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:42 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:42.281+0000 7faa313d5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:31:42.851 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:42 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:42.521+0000 7faa313d5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:31:42.851 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:42 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:42.653+0000 7faa313d5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:31:42.851 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:42 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:42.746+0000 7faa313d5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:31:42.851 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:42 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:42.861+0000 7faa313d5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:43.588 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:43 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:43.162+0000 7faa313d5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:31:43.873 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:43 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:43.772+0000 7faa313d5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:31:44.152 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:43 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:43.994+0000 7faa313d5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:31:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3543650832' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T12:31:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: Activating manager daemon y 2024-05-15T12:31:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: mgrmap e2: y(active, starting, since 0.00329608s) 2024-05-15T12:31:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:31:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: Manager daemon y is now available 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' 2024-05-15T12:31:44.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:44 smithi102 ceph-mon[24799]: from='mgr.14100 172.21.15.102:0/3366758508' entity='mgr.y' 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "eac36fa0-12b6-11ef-bc98-c7b262605968", 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T12:31:44.523 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T12:31:44.524 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T12:31:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T12:31:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T12:31:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T12:31:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T12:31:44.525 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T12:31:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T12:31:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T12:31:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T12:31:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T12:31:44.528 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T12:31:44.529 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T12:31:20.823853+0000", 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:31:44.530 INFO:teuthology.orchestra.run.smithi102.stdout:mgr not available, waiting (3/15)... 2024-05-15T12:31:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:46 smithi102 ceph-mon[24799]: mgrmap e3: y(active, since 1.00701s) 2024-05-15T12:31:47.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:47 smithi102 ceph-mon[24799]: mgrmap e4: y(active, since 2s) 2024-05-15T12:31:47.779 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:47.779 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:31:47.779 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsid": "eac36fa0-12b6-11ef-bc98-c7b262605968", 2024-05-15T12:31:47.779 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "health": { 2024-05-15T12:31:47.779 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-15T12:31:47.779 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-15T12:31:47.779 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 0 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "a" 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "quorum_age": 21, 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-15T12:31:47.780 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-15T12:31:47.781 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-15T12:31:47.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:47.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-15T12:31:47.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-15T12:31:47.784 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T12:31:47.785 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "restful" 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ], 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "modified": "2024-05-15T12:31:20.823853+0000", 2024-05-15T12:31:47.786 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-15T12:31:47.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout }, 2024-05-15T12:31:47.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-15T12:31:47.787 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:31:47.787 INFO:teuthology.orchestra.run.smithi102.stdout:mgr is available 2024-05-15T12:31:48.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:48 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2632600846' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-15T12:31:48.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:48.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [global] 2024-05-15T12:31:48.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout fsid = eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:31:48.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-15T12:31:48.959 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout [osd] 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-15T12:31:48.960 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling cephadm module... 2024-05-15T12:31:49.267 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:49 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2546034495' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-15T12:31:49.267 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:49 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2546034495' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-15T12:31:50.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:50 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1342494541' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-15T12:31:50.287 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ignoring --setuser ceph since I am not root 2024-05-15T12:31:50.287 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ignoring --setgroup ceph since I am not root 2024-05-15T12:31:50.287 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:50.176+0000 7f8bdaac1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:31:50.536 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:50.361+0000 7f8bdaac1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:50.537 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:50.466+0000 7f8bdaac1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:31:50.787 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:50.791+0000 7f8bdaac1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:31:51.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:51 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1342494541' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-15T12:31:51.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:51 smithi102 ceph-mon[24799]: mgrmap e5: y(active, since 6s) 2024-05-15T12:31:51.152 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:50.879+0000 7f8bdaac1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:31:51.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:31:51.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-15T12:31:51.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T12:31:51.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-05-15T12:31:51.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-15T12:31:51.863 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:31:51.864 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-05-15T12:31:51.864 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 5... 2024-05-15T12:31:52.221 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:52 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/573106490' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-15T12:31:52.221 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:52.104+0000 7f8bdaac1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:31:52.652 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:52.279+0000 7f8bdaac1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:52.652 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:52.453+0000 7f8bdaac1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:31:53.025 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:52.742+0000 7f8bdaac1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:31:53.026 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:52.835+0000 7f8bdaac1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:31:53.026 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:53 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:53.036+0000 7f8bdaac1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:31:53.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:53 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:53.228+0000 7f8bdaac1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:31:54.152 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:53 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:53.825+0000 7f8bdaac1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:31:54.534 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:54 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:54.245+0000 7f8bdaac1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:31:54.534 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:54 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:54.342+0000 7f8bdaac1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:31:54.535 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:54 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:54.545+0000 7f8bdaac1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:31:55.390 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:55 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:55.064+0000 7f8bdaac1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:31:55.391 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:55 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:55.156+0000 7f8bdaac1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:31:55.652 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:55 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:55.401+0000 7f8bdaac1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:31:55.652 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:55 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:55.535+0000 7f8bdaac1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:31:55.652 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:55 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:55.630+0000 7f8bdaac1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:31:56.034 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:55 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:55.746+0000 7f8bdaac1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:31:56.034 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:56 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:56.045+0000 7f8bdaac1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:31:56.901 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:56 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:56.643+0000 7f8bdaac1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:31:56.901 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:31:56 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:31:56.858+0000 7f8bdaac1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:31:57.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: Active manager daemon y restarted 2024-05-15T12:31:57.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: Activating manager daemon y 2024-05-15T12:31:57.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: osdmap e2: 0 total, 0 up, 0 in 2024-05-15T12:31:57.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: mgrmap e6: y(active, starting, since 0.00350462s) 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: Manager daemon y is now available 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:31:57.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:56 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:31:58.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:57 smithi102 ceph-mon[24799]: Found migration_current of "None". Setting to last migration. 2024-05-15T12:31:58.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:57 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:31:58.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:57 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:31:58.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:57 smithi102 ceph-mon[24799]: mgrmap e7: y(active, since 1.00535s) 2024-05-15T12:31:58.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:31:58.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-15T12:31:58.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-15T12:31:58.386 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:31:58.386 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 5 is available 2024-05-15T12:31:58.386 INFO:teuthology.orchestra.run.smithi102.stdout:Setting orchestrator backend to cephadm... 2024-05-15T12:31:59.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:58 smithi102 ceph-mon[24799]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-15T12:31:59.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:58 smithi102 ceph-mon[24799]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-15T12:31:59.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:58 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:31:59.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:31:58 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:00.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:00 smithi102 ceph-mon[24799]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:00.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:00 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:00.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:00 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:00.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:00 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:00.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:00 smithi102 ceph-mon[24799]: mgrmap e8: y(active, since 2s) 2024-05-15T12:32:00.718 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-15T12:32:00.718 INFO:teuthology.orchestra.run.smithi102.stdout:Generating ssh key... 2024-05-15T12:32:01.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:01 smithi102 ceph-mon[24799]: [15/May/2024:12:31:59] ENGINE Bus STARTING 2024-05-15T12:32:01.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:01 smithi102 ceph-mon[24799]: [15/May/2024:12:31:59] ENGINE Serving on https://172.21.15.102:7150 2024-05-15T12:32:01.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:01 smithi102 ceph-mon[24799]: [15/May/2024:12:31:59] ENGINE Serving on http://172.21.15.102:8765 2024-05-15T12:32:01.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:01 smithi102 ceph-mon[24799]: [15/May/2024:12:31:59] ENGINE Bus STARTED 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: Generating public/private rsa key pair. 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: Your identification has been saved in /tmp/tmpulubv1jg/key. 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: Your public key has been saved in /tmp/tmpulubv1jg/key.pub. 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: The key fingerprint is: 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: SHA256:XBsdlzd1S3qzUTGHRkIHrTzqx7OR8XIcHZS0dHvjeb0 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: The key's randomart image is: 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: +---[RSA 3072]----+ 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | .=+=XX| 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | . =Xo@| 2024-05-15T12:32:01.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | o..+ @o| 2024-05-15T12:32:01.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | . . o+ o.X| 2024-05-15T12:32:01.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | S .....++| 2024-05-15T12:32:01.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | . = .o| 2024-05-15T12:32:01.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | . .+ +E | 2024-05-15T12:32:01.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | . ++ | 2024-05-15T12:32:01.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: | ..o | 2024-05-15T12:32:01.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: +----[SHA256]-----+ 2024-05-15T12:32:02.218 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:02 smithi102 ceph-mon[24799]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:02.218 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:02 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:02.218 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:02 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:03.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:03 smithi102 ceph-mon[24799]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:03.092 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:03 smithi102 ceph-mon[24799]: Generating ssh key... 2024-05-15T12:32:03.094 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWHxrqBP4Dbg9OvObVy2VFrU3MqTCUzR2uzbYizFT6Ayb0SSQVwWiM9IOgALOL/wOYpvyBbMBhdB030Hjc6PXpjiLZuTSjtvE092mC05hXyvAM2wmQ/Lpl+vi4hy+uQzly4lhJPLykJC1YW2Iv3T5hdf605Fo7WCr9XkT6RdZd9WmFTmbK/xXlWjfIwlu/XEH7FHMFEdQJwFuQ4Kx4HniCbURlToYukQGy7Hi3EgXeHBVdgQJkewE3B9qVozTMMQclddEB0KRPm2N1ADTyUmTWSV691Uqx1hibQZazLFxrGRW30f6QIRyjrLJa9hSctaJ4XmxdEjBKakRaJbL/CwrTljQfRJiMrcFRQUUIjKenS8g1AyvYyy3LEeWeNEpOErXl/+VaJnT6B1THL0sdocPVjDEwrtReI76w03ZdqgQVDU1u1hm+FAODzUus6zIj16W7y3dPdstTFg30F0XwcOpSIqnQjYpC6y0Df5QEUTGVyPO0BXA1Q3qx/vcvJFAnm28= ceph-eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:32:03.094 INFO:teuthology.orchestra.run.smithi102.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-15T12:32:03.094 INFO:teuthology.orchestra.run.smithi102.stdout:Adding key to root@localhost authorized_keys... 2024-05-15T12:32:03.095 INFO:teuthology.orchestra.run.smithi102.stdout:Adding host smithi102... 2024-05-15T12:32:04.271 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:04 smithi102 ceph-mon[24799]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:05.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:05 smithi102 ceph-mon[24799]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "addr": "172.21.15.102", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:06.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:06 smithi102 ceph-mon[24799]: Deploying cephadm binary to smithi102 2024-05-15T12:32:08.303 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Added host 'smithi102' with addr '172.21.15.102' 2024-05-15T12:32:08.303 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mon service... 2024-05-15T12:32:09.002 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:08 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:09.002 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:08 smithi102 ceph-mon[24799]: Added host smithi102 2024-05-15T12:32:09.002 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:08 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:10.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:10 smithi102 ceph-mon[24799]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:10.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:10 smithi102 ceph-mon[24799]: Saving service mon spec with placement count:5 2024-05-15T12:32:10.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:10 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:10.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:10 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:10.496 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-15T12:32:10.496 INFO:teuthology.orchestra.run.smithi102.stdout:Deploying unmanaged mgr service... 2024-05-15T12:32:11.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:11 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:11.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:11 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:12.221 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-15T12:32:12.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:12 smithi102 ceph-mon[24799]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:12.862 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:12 smithi102 ceph-mon[24799]: Saving service mgr spec with placement count:2 2024-05-15T12:32:13.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:13.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:13.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:32:13.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' 2024-05-15T12:32:13.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:32:13.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: from='mgr.14120 172.21.15.102:0/3621610178' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]': finished 2024-05-15T12:32:13.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2578527319' entity='client.admin' 2024-05-15T12:32:13.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:13 smithi102 ceph-mon[24799]: Deploying daemon agent.smithi102 on smithi102 2024-05-15T12:32:14.500 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling the dashboard module... 2024-05-15T12:32:15.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:15 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1776348019' entity='client.admin' 2024-05-15T12:32:16.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:16 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/192563278' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-15T12:32:16.321 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:16 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ignoring --setuser ceph since I am not root 2024-05-15T12:32:16.321 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:16 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ignoring --setgroup ceph since I am not root 2024-05-15T12:32:16.321 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:16 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:16.200+0000 7ff7360ad200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:32:16.571 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:16 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:16.385+0000 7ff7360ad200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:32:16.571 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:16 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:16.488+0000 7ff7360ad200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:32:16.893 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:16 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:16.815+0000 7ff7360ad200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:32:16.893 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:16 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:16.903+0000 7ff7360ad200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:32:17.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:17 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/192563278' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-15T12:32:17.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:17 smithi102 ceph-mon[24799]: mgrmap e9: y(active, since 19s) 2024-05-15T12:32:17.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:32:17.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-15T12:32:17.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-15T12:32:17.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-05-15T12:32:17.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-15T12:32:17.841 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:32:17.841 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for the mgr to restart... 2024-05-15T12:32:17.842 INFO:teuthology.orchestra.run.smithi102.stdout:Waiting for mgr epoch 9... 2024-05-15T12:32:18.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:18 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/951069094' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-15T12:32:18.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:18 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:18.140+0000 7ff7360ad200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:32:18.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:18 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:18.318+0000 7ff7360ad200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:32:18.768 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:18 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:18.490+0000 7ff7360ad200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:32:18.769 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:18 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:18.779+0000 7ff7360ad200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:32:19.064 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:18 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:18.872+0000 7ff7360ad200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:32:19.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:19 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:19.074+0000 7ff7360ad200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:32:19.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:19 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:19.267+0000 7ff7360ad200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:32:20.152 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:19 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:19.867+0000 7ff7360ad200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:32:20.580 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:20 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:20.292+0000 7ff7360ad200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:32:20.581 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:20 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:20.389+0000 7ff7360ad200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:32:20.581 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:20 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:20.591+0000 7ff7360ad200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:32:21.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:21 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:21.111+0000 7ff7360ad200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:32:21.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:21 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:21.204+0000 7ff7360ad200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:32:21.786 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:21 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:21.449+0000 7ff7360ad200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:32:21.786 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:21 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:21.583+0000 7ff7360ad200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:32:21.786 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:21 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:21.679+0000 7ff7360ad200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:32:21.786 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:21 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:21.797+0000 7ff7360ad200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:32:22.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:22.096+0000 7ff7360ad200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:32:22.960 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:22.700+0000 7ff7360ad200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:32:22.960 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:32:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:32:22.917+0000 7ff7360ad200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:32:23.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: Active manager daemon y restarted 2024-05-15T12:32:23.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: Activating manager daemon y 2024-05-15T12:32:23.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: osdmap e3: 0 total, 0 up, 0 in 2024-05-15T12:32:23.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: mgrmap e10: y(active, starting, since 0.00294654s) 2024-05-15T12:32:23.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:32:23.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:32:23.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:32:23.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:32:23.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:32:23.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: Manager daemon y is now available 2024-05-15T12:32:23.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:23.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:23.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:24.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:23 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:32:24.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:23 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:32:24.137 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:23 smithi102 ceph-mon[24799]: mgrmap e11: y(active, since 1.00646s) 2024-05-15T12:32:24.776 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout { 2024-05-15T12:32:24.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-15T12:32:24.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-15T12:32:24.777 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout } 2024-05-15T12:32:24.777 INFO:teuthology.orchestra.run.smithi102.stdout:mgr epoch 9 is available 2024-05-15T12:32:24.777 INFO:teuthology.orchestra.run.smithi102.stdout:Generating a dashboard self-signed certificate... 2024-05-15T12:32:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: [15/May/2024:12:32:23] ENGINE Bus STARTING 2024-05-15T12:32:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: [15/May/2024:12:32:23] ENGINE Serving on http://172.21.15.102:8765 2024-05-15T12:32:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-15T12:32:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-15T12:32:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: [15/May/2024:12:32:23] ENGINE Serving on https://172.21.15.102:7150 2024-05-15T12:32:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: [15/May/2024:12:32:23] ENGINE Bus STARTED 2024-05-15T12:32:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:32:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:32:26.126 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-15T12:32:26.127 INFO:teuthology.orchestra.run.smithi102.stdout:Creating initial admin user... 2024-05-15T12:32:26.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:25 smithi102 ceph-mon[24799]: Deploying daemon agent.smithi102 on smithi102 2024-05-15T12:32:26.250 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:25 smithi102 ceph-mon[24799]: mgrmap e12: y(active, since 2s) 2024-05-15T12:32:26.250 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:25 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:26.250 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:25 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:27.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:26 smithi102 ceph-mon[24799]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:27.549 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8tIKSXQ6dB1xSpycAj5VfOD8gseTxxkKAXtwsENFqfzm3g.oeLMUW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715776347, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-15T12:32:27.549 INFO:teuthology.orchestra.run.smithi102.stdout:Fetching dashboard port number... 2024-05-15T12:32:28.194 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:28 smithi102 ceph-mon[24799]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:28.194 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:28.713 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stdout 8443 2024-05-15T12:32:28.722 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-15T12:32:28.722 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout disabled 2024-05-15T12:32:28.729 INFO:teuthology.orchestra.run.smithi102.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-15T12:32:28.729 INFO:teuthology.orchestra.run.smithi102.stdout:systemctl: stdout inactive 2024-05-15T12:32:28.729 INFO:teuthology.orchestra.run.smithi102.stdout:firewalld.service is not enabled 2024-05-15T12:32:28.730 INFO:teuthology.orchestra.run.smithi102.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-15T12:32:28.731 INFO:teuthology.orchestra.run.smithi102.stdout:Ceph Dashboard is now available at: 2024-05-15T12:32:28.732 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:28.732 INFO:teuthology.orchestra.run.smithi102.stdout: URL: https://smithi102.front.sepia.ceph.com:8443/ 2024-05-15T12:32:28.732 INFO:teuthology.orchestra.run.smithi102.stdout: User: admin 2024-05-15T12:32:28.732 INFO:teuthology.orchestra.run.smithi102.stdout: Password: 1fbs5bymdz 2024-05-15T12:32:28.732 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:28.732 INFO:teuthology.orchestra.run.smithi102.stdout:Saving cluster configuration to /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config directory 2024-05-15T12:32:28.733 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling autotune for osd_memory_target 2024-05-15T12:32:29.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:29 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/191473942' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-15T12:32:30.278 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:30 smithi102 ceph-mon[24799]: mgrmap e13: y(active, since 6s) 2024-05-15T12:32:31.184 INFO:teuthology.orchestra.run.smithi102.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout:Or, if you are only running a single cluster on this host: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: ceph telemetry on 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout:For more information see: 2024-05-15T12:32:31.185 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.186 INFO:teuthology.orchestra.run.smithi102.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-15T12:32:31.186 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:31.186 INFO:teuthology.orchestra.run.smithi102.stdout:Bootstrap complete. 2024-05-15T12:32:31.223 INFO:tasks.cephadm:Fetching config... 2024-05-15T12:32:31.224 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:32:31.224 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-15T12:32:31.239 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-15T12:32:31.239 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:32:31.239 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-15T12:32:31.294 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-15T12:32:31.294 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:32:31.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-15T12:32:31.475 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-15T12:32:31.475 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:32:31.475 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-15T12:32:31.530 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-15T12:32:31.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWHxrqBP4Dbg9OvObVy2VFrU3MqTCUzR2uzbYizFT6Ayb0SSQVwWiM9IOgALOL/wOYpvyBbMBhdB030Hjc6PXpjiLZuTSjtvE092mC05hXyvAM2wmQ/Lpl+vi4hy+uQzly4lhJPLykJC1YW2Iv3T5hdf605Fo7WCr9XkT6RdZd9WmFTmbK/xXlWjfIwlu/XEH7FHMFEdQJwFuQ4Kx4HniCbURlToYukQGy7Hi3EgXeHBVdgQJkewE3B9qVozTMMQclddEB0KRPm2N1ADTyUmTWSV691Uqx1hibQZazLFxrGRW30f6QIRyjrLJa9hSctaJ4XmxdEjBKakRaJbL/CwrTljQfRJiMrcFRQUUIjKenS8g1AyvYyy3LEeWeNEpOErXl/+VaJnT6B1THL0sdocPVjDEwrtReI76w03ZdqgQVDU1u1hm+FAODzUus6zIj16W7y3dPdstTFg30F0XwcOpSIqnQjYpC6y0Df5QEUTGVyPO0BXA1Q3qx/vcvJFAnm28= ceph-eac36fa0-12b6-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T12:32:31.685 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWHxrqBP4Dbg9OvObVy2VFrU3MqTCUzR2uzbYizFT6Ayb0SSQVwWiM9IOgALOL/wOYpvyBbMBhdB030Hjc6PXpjiLZuTSjtvE092mC05hXyvAM2wmQ/Lpl+vi4hy+uQzly4lhJPLykJC1YW2Iv3T5hdf605Fo7WCr9XkT6RdZd9WmFTmbK/xXlWjfIwlu/XEH7FHMFEdQJwFuQ4Kx4HniCbURlToYukQGy7Hi3EgXeHBVdgQJkewE3B9qVozTMMQclddEB0KRPm2N1ADTyUmTWSV691Uqx1hibQZazLFxrGRW30f6QIRyjrLJa9hSctaJ4XmxdEjBKakRaJbL/CwrTljQfRJiMrcFRQUUIjKenS8g1AyvYyy3LEeWeNEpOErXl/+VaJnT6B1THL0sdocPVjDEwrtReI76w03ZdqgQVDU1u1hm+FAODzUus6zIj16W7y3dPdstTFg30F0XwcOpSIqnQjYpC6y0Df5QEUTGVyPO0BXA1Q3qx/vcvJFAnm28= ceph-eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:32:31.697 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWHxrqBP4Dbg9OvObVy2VFrU3MqTCUzR2uzbYizFT6Ayb0SSQVwWiM9IOgALOL/wOYpvyBbMBhdB030Hjc6PXpjiLZuTSjtvE092mC05hXyvAM2wmQ/Lpl+vi4hy+uQzly4lhJPLykJC1YW2Iv3T5hdf605Fo7WCr9XkT6RdZd9WmFTmbK/xXlWjfIwlu/XEH7FHMFEdQJwFuQ4Kx4HniCbURlToYukQGy7Hi3EgXeHBVdgQJkewE3B9qVozTMMQclddEB0KRPm2N1ADTyUmTWSV691Uqx1hibQZazLFxrGRW30f6QIRyjrLJa9hSctaJ4XmxdEjBKakRaJbL/CwrTljQfRJiMrcFRQUUIjKenS8g1AyvYyy3LEeWeNEpOErXl/+VaJnT6B1THL0sdocPVjDEwrtReI76w03ZdqgQVDU1u1hm+FAODzUus6zIj16W7y3dPdstTFg30F0XwcOpSIqnQjYpC6y0Df5QEUTGVyPO0BXA1Q3qx/vcvJFAnm28= ceph-eac36fa0-12b6-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-15T12:32:31.727 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:31 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1146767963' entity='client.admin' 2024-05-15T12:32:31.738 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWHxrqBP4Dbg9OvObVy2VFrU3MqTCUzR2uzbYizFT6Ayb0SSQVwWiM9IOgALOL/wOYpvyBbMBhdB030Hjc6PXpjiLZuTSjtvE092mC05hXyvAM2wmQ/Lpl+vi4hy+uQzly4lhJPLykJC1YW2Iv3T5hdf605Fo7WCr9XkT6RdZd9WmFTmbK/xXlWjfIwlu/XEH7FHMFEdQJwFuQ4Kx4HniCbURlToYukQGy7Hi3EgXeHBVdgQJkewE3B9qVozTMMQclddEB0KRPm2N1ADTyUmTWSV691Uqx1hibQZazLFxrGRW30f6QIRyjrLJa9hSctaJ4XmxdEjBKakRaJbL/CwrTljQfRJiMrcFRQUUIjKenS8g1AyvYyy3LEeWeNEpOErXl/+VaJnT6B1THL0sdocPVjDEwrtReI76w03ZdqgQVDU1u1hm+FAODzUus6zIj16W7y3dPdstTFg30F0XwcOpSIqnQjYpC6y0Df5QEUTGVyPO0BXA1Q3qx/vcvJFAnm28= ceph-eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:32:31.751 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-15T12:32:32.043 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:33.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:34.853 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-15T12:32:34.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-15T12:32:35.058 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:34 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2519433668' entity='client.admin' 2024-05-15T12:32:35.059 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:34 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:35.059 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:34 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:32:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:39.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:39.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:39.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: Updating smithi102:/etc/ceph/ceph.conf 2024-05-15T12:32:39.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:39 smithi102 ceph-mon[24799]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:32:39.517 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-05-15T12:32:39.518 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:32:39.518 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-05-15T12:32:39.538 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:32:39.538 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:32:39.597 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-05-15T12:32:39.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch host add smithi173 2024-05-15T12:32:39.842 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:32:40.760 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:40 smithi102 ceph-mon[24799]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:32:40.760 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:40 smithi102 ceph-mon[24799]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:32:40.760 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:40.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:40.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:42 smithi102 ceph-mon[24799]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:42 smithi102 ceph-mon[24799]: Deploying cephadm binary to smithi173 2024-05-15T12:32:44.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:43 smithi102 ceph-mon[24799]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:45.494 INFO:teuthology.orchestra.run.smithi102.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-05-15T12:32:46.080 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-15T12:32:46.311 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:32:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:45 smithi102 ceph-mon[24799]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:47.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:47 smithi102 ceph-mon[24799]: Added host smithi173 2024-05-15T12:32:47.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:47.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:47.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:47.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:32:47.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:47.374 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:48.062 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:32:48.063 INFO:teuthology.orchestra.run.smithi102.stdout:[{"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-05-15T12:32:48.363 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:32:48.363 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:32:48.363 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:48.363 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:32:48.363 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:48.364 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:48.364 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:32:48.364 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]': finished 2024-05-15T12:32:48.619 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-15T12:32:48.619 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-15T12:32:48.855 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:32:49.309 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:32:49.309 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: Deploying daemon agent.smithi173 on smithi173 2024-05-15T12:32:49.309 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:32:49.309 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:49.309 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:49.310 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:50.108 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:50 smithi102 ceph-mon[24799]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:50.108 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:50.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:50.109 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:51.073 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-05-15T12:32:51.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:51 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3108137740' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-15T12:32:51.690 INFO:tasks.cephadm:Adding mon.a on smithi102 2024-05-15T12:32:51.690 INFO:tasks.cephadm:Adding mon.c on smithi102 2024-05-15T12:32:51.690 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-05-15T12:32:51.690 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b' 2024-05-15T12:32:52.009 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:32:52.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:52 smithi102 ceph-mon[24799]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:52.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:52 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3108137740' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-15T12:32:52.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:52 smithi102 ceph-mon[24799]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:32:53.350 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:53.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:53.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:53 smithi102 ceph-mon[24799]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:53.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:53.702 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-05-15T12:32:54.283 DEBUG:teuthology.orchestra.run.smithi102:mon.c> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.c.service 2024-05-15T12:32:54.286 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.b.service 2024-05-15T12:32:54.288 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T12:32:54.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-15T12:32:55.007 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:32:55.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:55.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b;count:3 2024-05-15T12:32:55.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:55.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:55.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:55.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:55.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:55.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:32:55.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:55.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: Deploying daemon mon.b on smithi173 2024-05-15T12:32:55.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:55.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:55.055 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:56.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:55 smithi102 ceph-mon[24799]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:58.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:32:57 smithi102 ceph-mon[24799]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:58.467 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:32:58.467 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"eac36fa0-12b6-11ef-bc98-c7b262605968","modified":"2024-05-15T12:31:17.803111Z","created":"2024-05-15T12:31:17.803111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-15T12:32:58.469 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-05-15T12:32:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 podman[26753]: 2024-05-15 12:32:58.715536658 +0000 UTC m=+0.439866931 container start 22e53ca710f1827f274d866088501c6056715a41b030ee622e922c0cdf2f735a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-15T12:32:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: set uid:gid to 167:167 (ceph:ceph) 2024-05-15T12:32:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev), process ceph-mon, pid 2 2024-05-15T12:32:58.729 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pidfile_write: ignore empty --pid-file 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: load: jerasure load: lrc 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: RocksDB version: 7.9.2 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Git sha 0 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Compile date 2024-05-14 19:11:07 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: DB SUMMARY 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: DB Session ID: N1AV4ADRY9UJC0C88QE6 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: CURRENT file: CURRENT 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: IDENTITY file: IDENTITY 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-05-15T12:32:58.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.error_if_exists: 0 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.create_if_missing: 0 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.paranoid_checks: 1 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.env: 0x55e860874b20 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.fs: PosixFileSystem 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.info_log: 0x55e861f1d900 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_file_opening_threads: 16 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.statistics: (nil) 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.use_fsync: 0 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_log_file_size: 0 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-15T12:32:58.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.keep_log_file_num: 1000 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.recycle_log_file_num: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.allow_fallocate: 1 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.allow_mmap_reads: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.allow_mmap_writes: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.use_direct_reads: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.create_missing_column_families: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.db_log_dir: 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.wal_dir: 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-15T12:32:58.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-15T12:32:58.999 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-15T12:32:58.999 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-15T12:32:58.999 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.advise_random_on_open: 1 2024-05-15T12:32:58.999 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.db_write_buffer_size: 0 2024-05-15T12:32:58.999 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.write_buffer_manager: 0x55e861ef1400 2024-05-15T12:32:58.999 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.rate_limiter: (nil) 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.wal_recovery_mode: 2 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.enable_thread_tracking: 0 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.enable_pipelined_write: 0 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.unordered_write: 0 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-15T12:32:59.000 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.row_cache: None 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.wal_filter: None 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.allow_ingest_behind: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.two_write_queues: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.manual_wal_flush: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.wal_compression: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.atomic_flush: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.log_readahead_size: 0 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-15T12:32:59.001 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.best_efforts_recovery: 0 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.allow_data_in_errors: 0 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.db_host_id: __hostname__ 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_background_jobs: 2 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_background_compactions: -1 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_subcompactions: 1 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_total_wal_size: 0 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-15T12:32:59.002 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_open_files: -1 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bytes_per_sync: 0 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_readahead_size: 0 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_background_flushes: -1 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Compression algorithms supported: 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kZSTD supported: 0 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kXpressCompression supported: 0 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kLZ4HCCompression supported: 1 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kZlibCompression supported: 1 2024-05-15T12:32:59.003 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kSnappyCompression supported: 1 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kLZ4Compression supported: 1 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: kBZip2Compression supported: 0 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.merge_operator: 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_filter: None 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_filter_factory: None 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.sst_partitioner_factory: None 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-15T12:32:59.004 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e861f1da00) 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_type: 0 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_index_type: 0 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_shortening: 1 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: checksum: 4 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: no_block_cache: 0 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache: 0x55e861f23090 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_options: 2024-05-15T12:32:59.005 INFO:journalctl@ceph.mon.b.smithi173.stdout: capacity : 536870912 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_shard_bits : 4 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: strict_capacity_limit : 0 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_compressed: (nil) 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: persistent_cache: (nil) 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size: 4096 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size_deviation: 10 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_restart_interval: 16 2024-05-15T12:32:59.006 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_block_restart_interval: 1 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: metadata_block_size: 4096 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: partition_filters: 0 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: use_delta_encoding: 1 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: filter_policy: bloomfilter 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: whole_key_filtering: 1 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: verify_compression: 0 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: format_version: 5 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_index_compression: 1 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_align: 0 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: max_auto_readahead_size: 262144 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: prepopulate_block_cache: 0 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: initial_auto_readahead_size: 8192 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-05-15T12:32:59.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.write_buffer_size: 33554432 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_write_buffer_number: 2 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression: NoCompression 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression: Disabled 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.prefix_extractor: nullptr 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.num_levels: 7 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-15T12:32:59.009 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.level: 32767 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.strategy: 0 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:32:59.010 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.enabled: false 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.target_file_size_base: 67108864 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-15T12:32:59.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.arena_block_size: 1048576 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.disable_auto_compactions: 0 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-15T12:32:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.table_properties_collectors: 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.inplace_update_support: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.bloom_locality: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.max_successive_merges: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.paranoid_file_checks: 0 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.force_consistency_checks: 1 2024-05-15T12:32:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.report_bg_io_stats: 0 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.ttl: 2592000 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.enable_blob_files: false 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.min_blob_size: 0 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.blob_file_size: 268435456 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-15T12:32:59.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.blob_file_starting_level: 0 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2c33eacd-eedb-44a6-ae27-c106c5250683 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776378746256, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776378746671, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715776378, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2c33eacd-eedb-44a6-ae27-c106c5250683", "db_session_id": "N1AV4ADRY9UJC0C88QE6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776378746751, "job": 1, "event": "recovery_finished"} 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e862016000 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: DB pointer 0x55e862000000 2024-05-15T12:32:59.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** DB Stats ** 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:32:59.016 INFO:journalctl@ceph.mon.b.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-15T12:32:59.017 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache BinnedLRUCache@0x55e861f23090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: starting mon.b rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(???) e0 preinit fsid eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).mds e1 new map 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: e1 2024-05-15T12:32:59.018 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout: legacy client fscid: -1 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout: No filesystems configured 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:32:59.019 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Added host smithi173 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.020 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]': finished 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Deploying daemon agent.smithi173 on smithi173 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:59.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/3108137740' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/3108137740' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:32:59.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b;count:3 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: Deploying daemon mon.b on smithi173 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:59.023 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:32:59.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 ceph-mon[26787]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-05-15T12:32:59.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 bash[26753]: 22e53ca710f1827f274d866088501c6056715a41b030ee622e922c0cdf2f735a 2024-05-15T12:32:59.024 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:32:58 smithi173 systemd[1]: Started Ceph mon.b for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:33:00.273 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T12:33:00.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-15T12:33:00.520 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:33:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:33:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: mon.a calling monitor election 2024-05-15T12:33:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: mon.b calling monitor election 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: fsmap 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: mgrmap e13: y(active, since 40s) 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: overall HEALTH_OK 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:04.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:04.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:04.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:04.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:04.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:33:04.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: mon.a calling monitor election 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: mon.b calling monitor election 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: fsmap 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: mgrmap e13: y(active, since 40s) 2024-05-15T12:33:04.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: overall HEALTH_OK 2024-05-15T12:33:04.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:04.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:04.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:04.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:04.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:05.061 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:04 smithi102 ceph-mon[24799]: Deploying daemon mon.c on smithi102 2024-05-15T12:33:05.061 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:04 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:05.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:04 smithi173 ceph-mon[26787]: Deploying daemon mon.c on smithi102 2024-05-15T12:33:05.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:04 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:05.243 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:33:05.243 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"eac36fa0-12b6-11ef-bc98-c7b262605968","modified":"2024-05-15T12:32:58.770992Z","created":"2024-05-15T12:31:17.803111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-15T12:33:05.245 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-05-15T12:33:05.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:05 smithi102 ceph-mon[24799]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:05.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:05 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/3106023009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:33:06.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:05 smithi173 ceph-mon[26787]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:06.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:05 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/3106023009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:33:06.830 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-15T12:33:06.831 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-15T12:33:07.074 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:33:07.376 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 podman[31318]: 2024-05-15 12:33:07.362729843 +0000 UTC m=+0.345174144 container start fa42c120c7419b9a5e32da87485fd49a0404f555dc890d3e7641a8d917d15958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS) 2024-05-15T12:33:07.376 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: set uid:gid to 167:167 (ceph:ceph) 2024-05-15T12:33:07.376 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev), process ceph-mon, pid 2 2024-05-15T12:33:07.376 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pidfile_write: ignore empty --pid-file 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: load: jerasure load: lrc 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: RocksDB version: 7.9.2 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Git sha 0 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Compile date 2024-05-14 19:11:07 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: DB SUMMARY 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: DB Session ID: UEM2FVVROXRSTBCE1P00 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: CURRENT file: CURRENT 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: IDENTITY file: IDENTITY 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-15T12:33:07.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.error_if_exists: 0 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.create_if_missing: 0 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.paranoid_checks: 1 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.env: 0x559050864b20 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.fs: PosixFileSystem 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.info_log: 0x5590528d1900 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_file_opening_threads: 16 2024-05-15T12:33:07.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.statistics: (nil) 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.use_fsync: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_log_file_size: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.keep_log_file_num: 1000 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.recycle_log_file_num: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.allow_fallocate: 1 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.allow_mmap_reads: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.allow_mmap_writes: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.use_direct_reads: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.create_missing_column_families: 0 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.db_log_dir: 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.wal_dir: 2024-05-15T12:33:07.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-15T12:33:07.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-15T12:33:07.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-15T12:33:07.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-15T12:33:07.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-15T12:33:07.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-15T12:33:07.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.advise_random_on_open: 1 2024-05-15T12:33:07.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.db_write_buffer_size: 0 2024-05-15T12:33:07.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.write_buffer_manager: 0x5590528a5400 2024-05-15T12:33:07.657 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.rate_limiter: (nil) 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.wal_recovery_mode: 2 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.enable_thread_tracking: 0 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.enable_pipelined_write: 0 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.unordered_write: 0 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-15T12:33:07.658 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.row_cache: None 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.wal_filter: None 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.allow_ingest_behind: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.two_write_queues: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.manual_wal_flush: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.wal_compression: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.atomic_flush: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.log_readahead_size: 0 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-15T12:33:07.659 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.best_efforts_recovery: 0 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.allow_data_in_errors: 0 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.db_host_id: __hostname__ 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_background_jobs: 2 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_background_compactions: -1 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_subcompactions: 1 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_total_wal_size: 0 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-15T12:33:07.660 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_open_files: -1 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bytes_per_sync: 0 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_readahead_size: 0 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_background_flushes: -1 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Compression algorithms supported: 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kZSTD supported: 0 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kXpressCompression supported: 0 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kLZ4HCCompression supported: 1 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kZlibCompression supported: 1 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kSnappyCompression supported: 1 2024-05-15T12:33:07.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kLZ4Compression supported: 1 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: kBZip2Compression supported: 0 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.merge_operator: 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_filter: None 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_filter_factory: None 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.sst_partitioner_factory: None 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5590528d1a00) 2024-05-15T12:33:07.662 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_type: 0 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_index_type: 0 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_shortening: 1 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: checksum: 4 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: no_block_cache: 0 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache: 0x5590528d7090 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_options: 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: capacity : 536870912 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_shard_bits : 4 2024-05-15T12:33:07.663 INFO:journalctl@ceph.mon.c.smithi102.stdout: strict_capacity_limit : 0 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_cache_compressed: (nil) 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: persistent_cache: (nil) 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size: 4096 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_size_deviation: 10 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_restart_interval: 16 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: index_block_restart_interval: 1 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: metadata_block_size: 4096 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: partition_filters: 0 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: use_delta_encoding: 1 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: filter_policy: bloomfilter 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: whole_key_filtering: 1 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: verify_compression: 0 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-05-15T12:33:07.664 INFO:journalctl@ceph.mon.c.smithi102.stdout: format_version: 5 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_index_compression: 1 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout: block_align: 0 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout: max_auto_readahead_size: 262144 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout: prepopulate_block_cache: 0 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout: initial_auto_readahead_size: 8192 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.write_buffer_size: 33554432 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_write_buffer_number: 2 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression: NoCompression 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression: Disabled 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.prefix_extractor: nullptr 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.num_levels: 7 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-15T12:33:07.665 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.level: 32767 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.strategy: 0 2024-05-15T12:33:07.666 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.enabled: false 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.target_file_size_base: 67108864 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-15T12:33:07.667 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.arena_block_size: 1048576 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.disable_auto_compactions: 0 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-15T12:33:07.668 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.table_properties_collectors: 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.inplace_update_support: 0 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.bloom_locality: 0 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.max_successive_merges: 0 2024-05-15T12:33:07.669 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.paranoid_file_checks: 0 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.force_consistency_checks: 1 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.report_bg_io_stats: 0 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.ttl: 2592000 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.enable_blob_files: false 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.min_blob_size: 0 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.blob_file_size: 268435456 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-15T12:33:07.670 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.blob_file_starting_level: 0 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f418f62a-46ec-4098-b935-85c1ecbf023f 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776387393320, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776387393705, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715776387, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f418f62a-46ec-4098-b935-85c1ecbf023f", "db_session_id": "UEM2FVVROXRSTBCE1P00", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715776387393784, "job": 1, "event": "recovery_finished"} 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5590529ca000 2024-05-15T12:33:07.671 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: DB pointer 0x5590529b4000 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** DB Stats ** 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-15T12:33:07.672 INFO:journalctl@ceph.mon.c.smithi102.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** Compaction Stats [default] ** 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-15T12:33:07.673 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache BinnedLRUCache@0x5590528d7090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: using public_addrv [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: starting mon.c rank -1 at public addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] at bind addrs [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(???) e0 preinit fsid eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).mds e1 new map 2024-05-15T12:33:07.674 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-15T12:33:07.675 INFO:journalctl@ceph.mon.c.smithi102.stdout: e1 2024-05-15T12:33:07.675 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-15T12:33:07.675 INFO:journalctl@ceph.mon.c.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-05-15T12:33:07.675 INFO:journalctl@ceph.mon.c.smithi102.stdout: legacy client fscid: -1 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout: No filesystems configured 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-15T12:33:07.676 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Added host smithi173 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:33:07.677 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]': finished 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Deploying daemon agent.smithi173 on smithi173 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.678 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.679 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/3108137740' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/3108137740' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Saving service mon spec with placement smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b;count:3 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:07.680 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Deploying daemon mon.b on smithi173 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.a calling monitor election 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:07.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.b calling monitor election 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: monmap e2: 2 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: fsmap 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mgrmap e13: y(active, since 40s) 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: overall HEALTH_OK 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: Deploying daemon mon.c on smithi102 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/3106023009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 ceph-mon[31347]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 bash[31318]: fa42c120c7419b9a5e32da87485fd49a0404f555dc890d3e7641a8d917d15958 2024-05-15T12:33:07.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:07 smithi102 systemd[1]: Started Ceph mon.c for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: mon.a calling monitor election 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: mon.b calling monitor election 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:12.732 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: fsmap 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: mgrmap e13: y(active, since 49s) 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: mon.c (rank 2) addr [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] is down (out of quorum) 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.733 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.734 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.734 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.734 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.734 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.734 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: mon.a calling monitor election 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: mon.b calling monitor election 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:12.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: fsmap 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: mgrmap e13: y(active, since 49s) 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: mon.c (rank 2) addr [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] is down (out of quorum) 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:12.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.217 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-05-15T12:33:13.218 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"eac36fa0-12b6-11ef-bc98-c7b262605968","modified":"2024-05-15T12:33:07.415519Z","created":"2024-05-15T12:31:17.803111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3301","nonce":0},{"type":"v1","addr":"172.21.15.102:6790","nonce":0}]},"addr":"172.21.15.102:6790/0","public_addr":"172.21.15.102:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-15T12:33:13.219 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: Updating smithi102:/etc/ceph/ceph.conf 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:33:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/756552483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:33:13.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: Updating smithi102:/etc/ceph/ceph.conf 2024-05-15T12:33:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:33:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:33:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/756552483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:33:13.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:13.950 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-15T12:33:13.950 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-15T12:33:14.281 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: mon.a calling monitor election 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: mon.b calling monitor election 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:14.681 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: fsmap 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: mgrmap e13: y(active, since 49s) 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: mon.c (rank 2) addr [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] is down (out of quorum) 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.682 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: Updating smithi102:/etc/ceph/ceph.conf 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:14.683 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/756552483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-15T12:33:14.684 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:15.270 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: mon.c calling monitor election 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: mon.c calling monitor election 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: mon.b calling monitor election 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: mon.a calling monitor election 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: fsmap 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: mgrmap e13: y(active, since 51s) 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: Cluster is now healthy 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: overall HEALTH_OK 2024-05-15T12:33:15.271 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[31347]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: mon.c calling monitor election 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: mon.c calling monitor election 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: mon.b calling monitor election 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: mon.a calling monitor election 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: fsmap 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: mgrmap e13: y(active, since 51s) 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-05-15T12:33:15.272 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: Cluster is now healthy 2024-05-15T12:33:15.273 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: overall HEALTH_OK 2024-05-15T12:33:15.273 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:15 smithi102 ceph-mon[24799]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:15.367 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: mon.c calling monitor election 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: mon.c calling monitor election 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: mon.b calling monitor election 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: mon.a calling monitor election 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: monmap e3: 3 mons at {a=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: fsmap 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: osdmap e4: 0 total, 0 up, 0 in 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: mgrmap e13: y(active, since 51s) 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: Cluster is now healthy 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: overall HEALTH_OK 2024-05-15T12:33:15.368 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:15 smithi173 ceph-mon[26787]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:16.346 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:16.352 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:16.352 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:33:17.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:17 smithi173 ceph-mon[26787]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:17.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.792 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[24799]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:17.793 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.793 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.793 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:17.793 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:17 smithi102 ceph-mon[31347]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:18.213 INFO:teuthology.orchestra.run.smithi102.stdout:# minimal ceph.conf for eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:33:18.213 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-05-15T12:33:18.213 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:33:18.213 INFO:teuthology.orchestra.run.smithi102.stdout: mon_host = [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.102:3301/0,v1:172.21.15.102:6790/0] 2024-05-15T12:33:18.725 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:18 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/859661620' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:18.725 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:18 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/859661620' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:18.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:18 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/859661620' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:19.549 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-15T12:33:19.549 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:33:19.550 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T12:33:19.576 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:33:19.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:33:19.642 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:33:19.643 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-05-15T12:33:19.678 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:33:19.678 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:33:19.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:19.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:19.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: Reconfiguring mon.c (monmap changed)... 2024-05-15T12:33:19.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:19.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:19.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:19.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: Reconfiguring daemon mon.c on smithi102 2024-05-15T12:33:19.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:19 smithi173 ceph-mon[26787]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:19.747 INFO:tasks.cephadm:Adding mgr.y on smithi102 2024-05-15T12:33:19.748 INFO:tasks.cephadm:Adding mgr.x on smithi173 2024-05-15T12:33:19.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply mgr '2;smithi102=y;smithi173=x' 2024-05-15T12:33:19.867 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:19.867 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:19.867 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: Reconfiguring mon.c (monmap changed)... 2024-05-15T12:33:19.867 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:19.867 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:19.867 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:19.867 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: Reconfiguring daemon mon.c on smithi102 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[31347]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: Reconfiguring mon.c (monmap changed)... 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: Reconfiguring daemon mon.c on smithi102 2024-05-15T12:33:19.868 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:19 smithi102 ceph-mon[24799]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:20.023 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:33:21.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[24799]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:21.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.295 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:21 smithi102 ceph-mon[31347]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:21.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:21 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:21 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:21 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:21.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:21 smithi173 ceph-mon[26787]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:21.770 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-05-15T12:33:22.282 DEBUG:teuthology.orchestra.run.smithi173:mgr.x> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.x.service 2024-05-15T12:33:22.285 INFO:tasks.cephadm:Deploying OSDs... 2024-05-15T12:33:22.285 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:33:22.285 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:33:22.305 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-05-15T12:33:22.305 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme1n1 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme1n1 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:33:19.449383305 +0000 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:30:28.136806017 +0000 2024-05-15T12:33:22.364 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:30:28.136806017 +0000 2024-05-15T12:33:22.365 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:30:28.131806114 +0000 2024-05-15T12:33:22.365 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-05-15T12:33:22.434 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:33:22.434 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:33:22.435 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000252203 s, 2.0 MB/s 2024-05-15T12:33:22.436 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: Reconfiguring mon.b (monmap changed)... 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: Reconfiguring daemon mon.b on smithi173 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:22.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: Saving service mgr spec with placement smithi102=y;smithi173=x;count:2 2024-05-15T12:33:22.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.495 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme2n1 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme2n1 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:33:19.460383090 +0000 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:30:28.260803590 +0000 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:30:28.260803590 +0000 2024-05-15T12:33:22.553 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:30:28.256803669 +0000 2024-05-15T12:33:22.553 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: Reconfiguring mon.b (monmap changed)... 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: Reconfiguring daemon mon.b on smithi173 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: Saving service mgr spec with placement smithi102=y;smithi173=x;count:2 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.616 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: Reconfiguring mon.b (monmap changed)... 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: Reconfiguring daemon mon.b on smithi173 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi102=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: Saving service mgr spec with placement smithi102=y;smithi173=x;count:2 2024-05-15T12:33:22.617 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:22.618 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:33:22.618 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:33:22.618 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000221663 s, 2.3 MB/s 2024-05-15T12:33:22.619 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-05-15T12:33:22.678 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme3n1 2024-05-15T12:33:22.734 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme3n1 2024-05-15T12:33:22.734 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:22.734 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-05-15T12:33:22.735 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:22.735 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:22.735 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:33:19.471382876 +0000 2024-05-15T12:33:22.735 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:30:28.392801007 +0000 2024-05-15T12:33:22.735 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:30:28.392801007 +0000 2024-05-15T12:33:22.735 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:30:28.388801086 +0000 2024-05-15T12:33:22.735 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-05-15T12:33:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:33:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:33:22.799 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000243399 s, 2.1 MB/s 2024-05-15T12:33:22.800 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-05-15T12:33:22.858 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/nvme4n1 2024-05-15T12:33:22.913 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/nvme4n1 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-05-15 12:33:19.482382661 +0000 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-05-15 12:30:28.523798444 +0000 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-05-15 12:30:28.523798444 +0000 2024-05-15T12:33:22.914 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-05-15 12:30:28.518798542 +0000 2024-05-15T12:33:22.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-05-15T12:33:22.977 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-05-15T12:33:22.977 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-05-15T12:33:22.977 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000236955 s, 2.2 MB/s 2024-05-15T12:33:22.979 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-05-15T12:33:23.035 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:33:23.035 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-05-15T12:33:23.052 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-05-15T12:33:23.052 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/nvme1n1 2024-05-15T12:33:23.107 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/nvme1n1 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:33:15.774173320 +0000 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:30:29.706860597 +0000 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:30:29.706860597 +0000 2024-05-15T12:33:23.108 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:30:29.701860703 +0000 2024-05-15T12:33:23.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-05-15T12:33:23.174 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:33:23.174 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:33:23.174 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000258338 s, 2.0 MB/s 2024-05-15T12:33:23.175 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-05-15T12:33:23.231 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/nvme2n1 2024-05-15T12:33:23.289 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/nvme2n1 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 776 Links: 1 Device type: 103,4 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:33:15.788173023 +0000 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:30:29.849857543 +0000 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:30:29.849857543 +0000 2024-05-15T12:33:23.290 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:30:29.844857650 +0000 2024-05-15T12:33:23.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-05-15T12:33:23.356 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:33:23.357 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:33:23.357 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000202177 s, 2.5 MB/s 2024-05-15T12:33:23.358 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-05-15T12:33:23.415 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/nvme3n1 2024-05-15T12:33:23.458 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:23 smithi173 ceph-mon[26787]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:23.470 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/nvme3n1 2024-05-15T12:33:23.470 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:23.470 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-05-15T12:33:23.470 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:23.470 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:23.471 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:33:15.800172767 +0000 2024-05-15T12:33:23.471 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:30:29.973854895 +0000 2024-05-15T12:33:23.471 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:30:29.973854895 +0000 2024-05-15T12:33:23.471 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:30:29.968855002 +0000 2024-05-15T12:33:23.471 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-05-15T12:33:23.534 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:33:23.534 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:33:23.534 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000285943 s, 1.8 MB/s 2024-05-15T12:33:23.535 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-05-15T12:33:23.591 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/nvme4n1 2024-05-15T12:33:23.647 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/nvme4n1 2024-05-15T12:33:23.648 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-05-15T12:33:23.648 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-05-15T12:33:23.648 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-05-15T12:33:23.648 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-05-15T12:33:23.648 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-05-15 12:33:15.811172533 +0000 2024-05-15T12:33:23.649 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-05-15 12:30:30.106852055 +0000 2024-05-15T12:33:23.649 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-05-15 12:30:30.106852055 +0000 2024-05-15T12:33:23.649 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-05-15 12:30:30.101852162 +0000 2024-05-15T12:33:23.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-05-15T12:33:23.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:23 smithi102 ceph-mon[24799]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:23.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:23 smithi102 ceph-mon[31347]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:23.712 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-05-15T12:33:23.713 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-05-15T12:33:23.713 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000208134 s, 2.5 MB/s 2024-05-15T12:33:23.714 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-05-15T12:33:23.771 INFO:tasks.cephadm:Deploying osd.0 on smithi102 with /dev/nvme4n1... 2024-05-15T12:33:23.771 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1 2024-05-15T12:33:24.017 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:33:24.393 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:24.394 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[24799]: Deploying daemon mgr.x on smithi173 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:24.395 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:24 smithi102 ceph-mon[31347]: Deploying daemon mgr.x on smithi173 2024-05-15T12:33:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:24.703 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:24.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:33:24.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-15T12:33:24.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:33:24.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:24.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:24 smithi173 ceph-mon[26787]: Deploying daemon mgr.x on smithi173 2024-05-15T12:33:25.649 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:25 smithi173 ceph-mon[26787]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:25.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:25 smithi102 ceph-mon[24799]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:25.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:25 smithi102 ceph-mon[31347]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme4n1 2024-05-15T12:33:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:33:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-05-15T12:33:25.766 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-05-15T12:33:25.766 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-05-15T12:33:25.766 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125317 s, 837 MB/s 2024-05-15T12:33:25.766 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-05-15T12:33:26.120 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:25.941+0000 7f2d86a4c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:33:26.120 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:26.130+0000 7f2d86a4c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:33:26.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi102:/dev/nvme4n1 2024-05-15T12:33:26.428 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:26.238+0000 7f2d86a4c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:33:26.649 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:33:26.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:26.565+0000 7f2d86a4c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:33:26.736 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:26.650+0000 7f2d86a4c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:33:26.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:33:26.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:27.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:27.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:27.105 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:27.105 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:27.105 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-15T12:33:27.105 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-15T12:33:27.105 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:27.917 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:27.893+0000 7f2d86a4c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:33:28.200 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[24799]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:33:28.200 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[24799]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T12:33:28.201 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[24799]: Reconfiguring daemon mgr.y on smithi102 2024-05-15T12:33:28.201 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[24799]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:28.201 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[31347]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:33:28.201 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[31347]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T12:33:28.201 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[31347]: Reconfiguring daemon mgr.y on smithi102 2024-05-15T12:33:28.201 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:27 smithi102 ceph-mon[31347]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:28.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:28.084+0000 7f2d86a4c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:33:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:27 smithi173 ceph-mon[26787]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:33:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:27 smithi173 ceph-mon[26787]: Reconfiguring mgr.y (unknown last config time)... 2024-05-15T12:33:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:27 smithi173 ceph-mon[26787]: Reconfiguring daemon mgr.y on smithi102 2024-05-15T12:33:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:27 smithi173 ceph-mon[26787]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:28.601 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:28.289+0000 7f2d86a4c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:33:28.601 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:28.611+0000 7f2d86a4c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:33:28.905 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:28.704+0000 7f2d86a4c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:28.915+0000 7f2d86a4c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:29.111+0000 7f2d86a4c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:28 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:28 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:28 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:28 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:28 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:29.210 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:28 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:29.211 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:28 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:29.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:29.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:29.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:28 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:29.973 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:29.737+0000 7f2d86a4c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:30.159+0000 7f2d86a4c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:30.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:30.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:30.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:30.259 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:30.260 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:30.261 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:30.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:30.255+0000 7f2d86a4c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:33:30.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:30.456+0000 7f2d86a4c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:33:31.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:30 smithi102 ceph-mon[24799]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:31.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:30 smithi102 ceph-mon[31347]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:31.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:30 smithi173 ceph-mon[26787]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:31.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:30.979+0000 7f2d86a4c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:33:31.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:31.070+0000 7f2d86a4c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:33:31.641 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:31.311+0000 7f2d86a4c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:33:31.641 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:31.443+0000 7f2d86a4c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:33:31.641 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:31.537+0000 7f2d86a4c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:33:31.641 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:31.651+0000 7f2d86a4c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:33:32.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:31 smithi173 ceph-mon[26787]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:32.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:31.957+0000 7f2d86a4c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:33:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:31 smithi102 ceph-mon[24799]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:32.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:31 smithi102 ceph-mon[31347]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:32.978 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:32.555+0000 7f2d86a4c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:33:32.979 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:33:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:33:32.769+0000 7f2d86a4c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/3071715817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a157badc-d604-423e-80fd-7205437c36de"}]: dispatch 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/3071715817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a157badc-d604-423e-80fd-7205437c36de"}]': finished 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: Standby manager daemon x started 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:33:33.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:32 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3071715817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a157badc-d604-423e-80fd-7205437c36de"}]: dispatch 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3071715817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a157badc-d604-423e-80fd-7205437c36de"}]': finished 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: Standby manager daemon x started 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:33:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/3071715817' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a157badc-d604-423e-80fd-7205437c36de"}]: dispatch 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/3071715817' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a157badc-d604-423e-80fd-7205437c36de"}]': finished 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: osdmap e5: 1 total, 0 up, 1 in 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: Standby manager daemon x started 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:32 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/3795594063' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:33:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[31347]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[31347]: mgrmap e14: y(active, since 70s), standbys: x 2024-05-15T12:33:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:33:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2872239689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:33:34.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[24799]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:34.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[24799]: mgrmap e14: y(active, since 70s), standbys: x 2024-05-15T12:33:34.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:33:34.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:34 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2872239689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:33:34.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:34 smithi173 ceph-mon[26787]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:34.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:34 smithi173 ceph-mon[26787]: mgrmap e14: y(active, since 70s), standbys: x 2024-05-15T12:33:34.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:34 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:33:34.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:34 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2872239689' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:33:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:35 smithi102 ceph-mon[24799]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:35.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:35 smithi102 ceph-mon[31347]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:35.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:35 smithi173 ceph-mon[26787]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:38.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:37 smithi102 ceph-mon[24799]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:38.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:37 smithi102 ceph-mon[31347]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:38.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:37 smithi173 ceph-mon[26787]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:40.161 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:39 smithi102 ceph-mon[24799]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:40.161 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:39 smithi102 ceph-mon[31347]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:40.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:39 smithi173 ceph-mon[26787]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T12:33:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T12:33:41.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:41.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-15T12:33:41.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:41.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:41.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:42.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:41 smithi173 ceph-mon[26787]: Deploying daemon osd.0 on smithi102 2024-05-15T12:33:42.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:41 smithi173 ceph-mon[26787]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:42.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:41 smithi102 ceph-mon[24799]: Deploying daemon osd.0 on smithi102 2024-05-15T12:33:42.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:41 smithi102 ceph-mon[24799]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:42.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:41 smithi102 ceph-mon[31347]: Deploying daemon osd.0 on smithi102 2024-05-15T12:33:42.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:41 smithi102 ceph-mon[31347]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:43 smithi102 ceph-mon[24799]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:44.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:43 smithi102 ceph-mon[31347]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:44.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:43 smithi173 ceph-mon[26787]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:46.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:45 smithi173 ceph-mon[26787]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:45 smithi102 ceph-mon[24799]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:46.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:45 smithi102 ceph-mon[31347]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:48.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:48 smithi173 ceph-mon[26787]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:48.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:48 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:48 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:48 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:48.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:48 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:48.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:48 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:48.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:48 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[24799]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[31347]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:48.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:48 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[31347]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:49 smithi102 ceph-mon[24799]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:49.702 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:49 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.702 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:49 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.702 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:49 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:49.702 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:49 smithi173 ceph-mon[26787]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:50.140 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 0 on host 'smithi102' 2024-05-15T12:33:50.738 DEBUG:teuthology.orchestra.run.smithi102:osd.0> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.0.service 2024-05-15T12:33:50.740 INFO:tasks.cephadm:Deploying osd.1 on smithi102 with /dev/nvme3n1... 2024-05-15T12:33:50.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme3n1 2024-05-15T12:33:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:33:51.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi102 to 15040M 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[24799]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi102 to 15040M 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:51 smithi102 ceph-mon[31347]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:51.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-15T12:33:51.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:33:51.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:33:51.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi102 to 15040M 2024-05-15T12:33:51.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:51.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:33:51.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:51.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:51 smithi173 ceph-mon[26787]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:52.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[24799]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T12:33:52.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[24799]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T12:33:52.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[24799]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:33:52.286 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:52.286 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[31347]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T12:33:52.287 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[31347]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T12:33:52.287 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[31347]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:33:52.287 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:52.287 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:33:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0[35771]: 2024-05-15T12:33:52.122+0000 7f7be5001700 -1 osd.0 0 waiting for initial osdmap 2024-05-15T12:33:52.287 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:33:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0[35771]: 2024-05-15T12:33:52.125+0000 7f7be0df4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:33:52.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:52 smithi173 ceph-mon[26787]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-15T12:33:52.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:52 smithi173 ceph-mon[26787]: osdmap e6: 1 total, 0 up, 1 in 2024-05-15T12:33:52.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:52 smithi173 ceph-mon[26787]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:33:52.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:52.835 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme3n1 2024-05-15T12:33:52.835 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:33:52.835 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-05-15T12:33:52.835 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-05-15T12:33:52.835 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-05-15T12:33:52.835 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012967 s, 809 MB/s 2024-05-15T12:33:52.835 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-05-15T12:33:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:33:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:33:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[24799]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:33:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[24799]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T12:33:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[24799]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:53.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:33:53.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:33:53.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[31347]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:33:53.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[31347]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T12:33:53.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:53.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:53.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:53 smithi102 ceph-mon[31347]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:53.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi102:/dev/nvme3n1 2024-05-15T12:33:53.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:53 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:33:53.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:53 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:33:53.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:53 smithi173 ceph-mon[26787]: from='osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:33:53.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:53 smithi173 ceph-mon[26787]: osdmap e7: 1 total, 0 up, 1 in 2024-05-15T12:33:53.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:53.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:53.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:53 smithi173 ceph-mon[26787]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-15T12:33:53.721 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:33:54.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:54.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[24799]: osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593] boot 2024-05-15T12:33:54.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[24799]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T12:33:54.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:54.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:54.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[31347]: osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593] boot 2024-05-15T12:33:54.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[31347]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T12:33:54.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:54 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:54.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:54 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:54.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:54 smithi173 ceph-mon[26787]: osd.0 [v2:172.21.15.102:6802/4160180593,v1:172.21.15.102:6803/4160180593] boot 2024-05-15T12:33:54.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:54 smithi173 ceph-mon[26787]: osdmap e8: 1 total, 1 up, 1 in 2024-05-15T12:33:54.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:54 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:33:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:55 smithi102 ceph-mon[31347]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T12:33:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:55 smithi102 ceph-mon[31347]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:33:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:55 smithi102 ceph-mon[24799]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T12:33:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:55 smithi102 ceph-mon[24799]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:33:55.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:55 smithi173 ceph-mon[26787]: osdmap e9: 1 total, 1 up, 1 in 2024-05-15T12:33:55.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:55 smithi173 ceph-mon[26787]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:33:56.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:33:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:33:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:56.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:33:56.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:33:56.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:56.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:33:56.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:33:56.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:33:57.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[31347]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:57.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi102 2024-05-15T12:33:57.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[31347]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:33:57.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[24799]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:57.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi102 2024-05-15T12:33:57.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:57 smithi102 ceph-mon[24799]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:33:57.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:57 smithi173 ceph-mon[26787]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:33:57.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:57 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:57 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:57 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi102 2024-05-15T12:33:57.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:57 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:33:57.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:57 smithi173 ceph-mon[26787]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:00.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:59 smithi173 ceph-mon[26787]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:00.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:59 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/1944324292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]: dispatch 2024-05-15T12:34:00.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:59 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]: dispatch 2024-05-15T12:34:00.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:59 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]': finished 2024-05-15T12:34:00.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:59 smithi173 ceph-mon[26787]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T12:34:00.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:33:59 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:00.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[24799]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:00.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1944324292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]: dispatch 2024-05-15T12:34:00.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]: dispatch 2024-05-15T12:34:00.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]': finished 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[24799]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[31347]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/1944324292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]: dispatch 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]: dispatch 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cfe8e1f-978d-4e11-94f3-094a13ac15fa"}]': finished 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[31347]: osdmap e10: 2 total, 1 up, 2 in 2024-05-15T12:34:00.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:33:59 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:01.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:00 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2199754784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:01.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:00 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2199754784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:01.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:00 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2199754784' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:02.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:01 smithi173 ceph-mon[26787]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:02.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:01 smithi102 ceph-mon[24799]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:02.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:01 smithi102 ceph-mon[31347]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:04.228 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:03 smithi102 ceph-mon[24799]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:04.276 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:04 smithi102 ceph-mon[31347]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:04.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:03 smithi173 ceph-mon[26787]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:06.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:06 smithi102 ceph-mon[24799]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:06.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:06 smithi102 ceph-mon[31347]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:06.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:06 smithi173 ceph-mon[26787]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:07.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:07 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T12:34:07.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:07 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:07.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T12:34:07.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:07.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:07 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-15T12:34:07.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:07 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:08.081 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:08 smithi173 ceph-mon[26787]: Deploying daemon osd.1 on smithi102 2024-05-15T12:34:08.081 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:08 smithi173 ceph-mon[26787]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:08.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:08 smithi102 ceph-mon[24799]: Deploying daemon osd.1 on smithi102 2024-05-15T12:34:08.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:08 smithi102 ceph-mon[24799]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:08.148 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:08 smithi102 ceph-mon[31347]: Deploying daemon osd.1 on smithi102 2024-05-15T12:34:08.149 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:08 smithi102 ceph-mon[31347]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:09.854 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.854 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.854 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.854 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[31347]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:09.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.855 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:09 smithi102 ceph-mon[24799]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:09.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:09 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:09 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:09 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:09.862 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:09 smithi173 ceph-mon[26787]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:12.242 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:11 smithi102 ceph-mon[24799]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:12.242 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:11 smithi102 ceph-mon[31347]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:12.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:11 smithi173 ceph-mon[26787]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[24799]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:13.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:13.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:13.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:13.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:13 smithi102 ceph-mon[31347]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:13.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:13 smithi173 ceph-mon[26787]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:15.393 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 1 on host 'smithi102' 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:34:16.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[24799]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:34:16.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:15 smithi102 ceph-mon[31347]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:34:16.060 DEBUG:teuthology.orchestra.run.smithi102:osd.1> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.1.service 2024-05-15T12:34:16.063 INFO:tasks.cephadm:Deploying osd.2 on smithi102 with /dev/nvme2n1... 2024-05-15T12:34:16.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme2n1 2024-05-15T12:34:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:16.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:16.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:16.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:34:16.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:15 smithi173 ceph-mon[26787]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-15T12:34:17.029 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi102 to 7520M 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:17.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi102 to 7520M 2024-05-15T12:34:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T12:34:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T12:34:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:17.257 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi102 to 7520M 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: osdmap e11: 2 total, 1 up, 2 in 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:17.258 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:17.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:17.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:17.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:17.259 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:17.652 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:17 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1[39163]: 2024-05-15T12:34:17.429+0000 7f699e770700 -1 osd.1 0 waiting for initial osdmap 2024-05-15T12:34:17.652 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:17 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1[39163]: 2024-05-15T12:34:17.434+0000 7f699a563700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:34:18.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:17 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi102 2024-05-15T12:34:18.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:17 smithi173 ceph-mon[26787]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:18.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:17 smithi173 ceph-mon[26787]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:34:18.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:17 smithi173 ceph-mon[26787]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T12:34:18.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:18.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:18.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi102 2024-05-15T12:34:18.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[24799]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:18.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[24799]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:34:18.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[24799]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T12:34:18.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:18.284 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:18.284 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi102 2024-05-15T12:34:18.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[31347]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:18.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[31347]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:34:18.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[31347]: osdmap e12: 2 total, 1 up, 2 in 2024-05-15T12:34:18.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:18.285 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:18.827 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme2n1 2024-05-15T12:34:18.827 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:34:18.827 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-05-15T12:34:18.827 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-05-15T12:34:18.827 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-05-15T12:34:18.827 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131475 s, 798 MB/s 2024-05-15T12:34:18.827 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-05-15T12:34:19.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi102:/dev/nvme2n1 2024-05-15T12:34:19.740 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:34:19.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088] boot 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:19 smithi173 ceph-mon[26787]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088] boot 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:19.863 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[31347]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: osd.1 [v2:172.21.15.102:6810/838755088,v1:172.21.15.102:6811/838755088] boot 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: osdmap e13: 2 total, 2 up, 2 in 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:19.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:19.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:19.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:19.865 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:19 smithi102 ceph-mon[24799]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-15T12:34:20.688 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:20 smithi173 ceph-mon[26787]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T12:34:20.869 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:20 smithi102 ceph-mon[24799]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T12:34:20.870 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:20 smithi102 ceph-mon[31347]: osdmap e14: 2 total, 2 up, 2 in 2024-05-15T12:34:21.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:21 smithi173 ceph-mon[26787]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:21.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:21 smithi102 ceph-mon[24799]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:21.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:21 smithi102 ceph-mon[31347]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:22.543 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[24799]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:34:22.544 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:34:22.544 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:34:22.544 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:22.544 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[31347]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:34:22.544 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:34:22.544 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:34:22.544 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:22.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:22 smithi173 ceph-mon[26787]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:34:22.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:34:22.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:34:22.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:23.688 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:23 smithi102 ceph-mon[31347]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:23.689 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:23 smithi102 ceph-mon[24799]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:23.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:23 smithi173 ceph-mon[26787]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:26.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:25 smithi173 ceph-mon[26787]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:25 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/3726155689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]: dispatch 2024-05-15T12:34:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:25 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]: dispatch 2024-05-15T12:34:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:25 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]': finished 2024-05-15T12:34:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:25 smithi173 ceph-mon[26787]: osdmap e15: 3 total, 2 up, 3 in 2024-05-15T12:34:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:25 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[24799]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3726155689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]: dispatch 2024-05-15T12:34:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]: dispatch 2024-05-15T12:34:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]': finished 2024-05-15T12:34:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[24799]: osdmap e15: 3 total, 2 up, 3 in 2024-05-15T12:34:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:26.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[31347]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:26.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/3726155689' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]: dispatch 2024-05-15T12:34:26.405 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]: dispatch 2024-05-15T12:34:26.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f49f2eb3-557b-456c-91ab-7098fd58995a"}]': finished 2024-05-15T12:34:26.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[31347]: osdmap e15: 3 total, 2 up, 3 in 2024-05-15T12:34:26.406 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:25 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:27.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:26 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2093144487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:27.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:26 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2093144487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:27.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:26 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2093144487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:27 smithi173 ceph-mon[26787]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:28.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:27 smithi102 ceph-mon[24799]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:28.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:27 smithi102 ceph-mon[31347]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:29.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:29 smithi173 ceph-mon[26787]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[31347]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:29.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:29 smithi102 ceph-mon[24799]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:32.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:31 smithi173 ceph-mon[26787]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:32.314 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:31 smithi102 ceph-mon[31347]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:32.314 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:31 smithi102 ceph-mon[24799]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:33.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T12:34:33.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:33.268 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T12:34:33.268 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:33.268 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:33 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-15T12:34:33.268 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:33 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:34.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:34 smithi173 ceph-mon[26787]: Deploying daemon osd.2 on smithi102 2024-05-15T12:34:34.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:34 smithi173 ceph-mon[26787]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:34.293 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:34 smithi102 ceph-mon[24799]: Deploying daemon osd.2 on smithi102 2024-05-15T12:34:34.294 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:34 smithi102 ceph-mon[24799]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:34.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:34 smithi102 ceph-mon[31347]: Deploying daemon osd.2 on smithi102 2024-05-15T12:34:34.294 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:34 smithi102 ceph-mon[31347]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:36.224 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:36 smithi102 ceph-mon[24799]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:36.225 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:36 smithi102 ceph-mon[31347]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:36.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:36 smithi173 ceph-mon[26787]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:38.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:38 smithi102 ceph-mon[24799]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:38.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:38 smithi102 ceph-mon[31347]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:38.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:38 smithi173 ceph-mon[26787]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:40.086 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.087 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.088 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.089 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:39 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:39 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:40.961 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:40 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:41.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:40 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:41.873 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:41 smithi102 ceph-mon[31347]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:41.874 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:41 smithi102 ceph-mon[24799]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:42.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:41 smithi173 ceph-mon[26787]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:42.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:42 smithi102 ceph-mon[24799]: from='osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:34:42.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:42 smithi102 ceph-mon[24799]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:34:42.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:42 smithi102 ceph-mon[31347]: from='osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:34:42.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:42 smithi102 ceph-mon[31347]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:34:43.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:42 smithi173 ceph-mon[26787]: from='osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:34:43.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:42 smithi173 ceph-mon[26787]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-15T12:34:43.931 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 2 on host 'smithi102' 2024-05-15T12:34:44.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[24799]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[24799]: osdmap e16: 3 total, 2 up, 3 in 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[24799]: from='osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[24799]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[24799]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[31347]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[31347]: osdmap e16: 3 total, 2 up, 3 in 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[31347]: from='osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[31347]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:44.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:43 smithi102 ceph-mon[31347]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:44.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:43 smithi173 ceph-mon[26787]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-15T12:34:44.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:43 smithi173 ceph-mon[26787]: osdmap e16: 3 total, 2 up, 3 in 2024-05-15T12:34:44.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:43 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:44.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:43 smithi173 ceph-mon[26787]: from='osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:44.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:43 smithi173 ceph-mon[26787]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:34:44.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:43 smithi173 ceph-mon[26787]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:44.498 DEBUG:teuthology.orchestra.run.smithi102:osd.2> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.2.service 2024-05-15T12:34:44.501 INFO:tasks.cephadm:Deploying osd.3 on smithi102 with /dev/nvme1n1... 2024-05-15T12:34:44.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme1n1 2024-05-15T12:34:44.774 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:34:45.098 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:34:45.098 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T12:34:45.098 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:45.098 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:45.098 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi102 to 5013M 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:45.099 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi102 to 5013M 2024-05-15T12:34:45.100 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:45.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:45.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: osdmap e17: 3 total, 2 up, 3 in 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi102 to 5013M 2024-05-15T12:34:45.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:45.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:45.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:34:45.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:46.070 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300] boot 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: osdmap e18: 3 total, 3 up, 3 in 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300] boot 2024-05-15T12:34:46.071 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: osdmap e18: 3 total, 3 up, 3 in 2024-05-15T12:34:46.072 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:46.072 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:46.072 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:34:46.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:34:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:34:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: osd.2 [v2:172.21.15.102:6818/1924240300,v1:172.21.15.102:6819/1924240300] boot 2024-05-15T12:34:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: osdmap e18: 3 total, 3 up, 3 in 2024-05-15T12:34:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:34:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-15T12:34:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:34:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/nvme1n1 2024-05-15T12:34:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:34:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-05-15T12:34:46.625 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-05-15T12:34:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-05-15T12:34:46.625 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012904 s, 813 MB/s 2024-05-15T12:34:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-05-15T12:34:46.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:46 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T12:34:46.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:46 smithi102 ceph-mon[24799]: osdmap e19: 3 total, 3 up, 3 in 2024-05-15T12:34:46.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:46 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:34:46.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:46 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T12:34:46.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:46 smithi102 ceph-mon[31347]: osdmap e19: 3 total, 3 up, 3 in 2024-05-15T12:34:46.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:46 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:34:47.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:46 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-15T12:34:47.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:46 smithi173 ceph-mon[26787]: osdmap e19: 3 total, 3 up, 3 in 2024-05-15T12:34:47.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:46 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-15T12:34:47.276 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi102:/dev/nvme1n1 2024-05-15T12:34:47.519 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:34:47.619 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44197]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-05-15T12:34:47.619 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44197]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:47.619 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44197]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:47.619 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44197]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:47.870 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44253]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-05-15T12:34:47.870 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44253]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:47.870 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44253]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:47.870 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44253]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:48.146 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44264]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme3n1 2024-05-15T12:34:48.147 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44264]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:48.147 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44264]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:48.147 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:47 smithi102 sudo[44264]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:48.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:34:48.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[31347]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T12:34:48.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[31347]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:48.147 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:48.147 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:34:48.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[24799]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T12:34:48.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[24799]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:48.148 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:47 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:48.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:47 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-15T12:34:48.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:47 smithi173 ceph-mon[26787]: osdmap e20: 3 total, 3 up, 3 in 2024-05-15T12:34:48.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:47 smithi173 ceph-mon[26787]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:48.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:47 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:48.401 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44278]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme3n1 2024-05-15T12:34:48.402 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44278]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:48.402 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44278]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:48.402 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44278]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:48.673 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44282]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2024-05-15T12:34:48.673 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:48.673 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44282]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:48.673 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44282]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:48.925 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44304]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme2n1 2024-05-15T12:34:48.925 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44304]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:48.925 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44304]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:48.925 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44304]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:49.205 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44313]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:34:49.205 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44313]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:49.205 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44313]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:49.205 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:48 smithi102 sudo[44313]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:49.240 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 ceph-mon[26787]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T12:34:49.240 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:49.240 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:49.240 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 ceph-mon[26787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:49.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30584]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:34:49.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30584]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:49.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30584]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[31347]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T12:34:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:49.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[31347]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:49.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 sudo[44338]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:34:49.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 sudo[44338]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:49.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 sudo[44338]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 sudo[44338]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[24799]: osdmap e21: 3 total, 3 up, 3 in 2024-05-15T12:34:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:49.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:49 smithi102 ceph-mon[24799]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:49.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30584]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:49.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30607]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:34:49.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30607]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:49.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30607]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:49.789 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:49 smithi173 sudo[30607]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:50.277 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44463]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-15T12:34:50.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:50.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44463]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:50.278 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44463]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:50 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:50.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:50.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44484]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44484]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44484]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 sudo[44484]: pam_unix(sudo:session): session closed for user root 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: mgrmap e15: y(active, since 2m), standbys: x 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:50.655 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:34:50.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:34:50.656 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:50 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:34:51.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:51 smithi173 ceph-mon[26787]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:51.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:51 smithi173 ceph-mon[26787]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:51.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:51 smithi173 ceph-mon[26787]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:51.583 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:51 smithi102 ceph-mon[24799]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:51.583 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:51 smithi102 ceph-mon[24799]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:51.583 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:51 smithi102 ceph-mon[24799]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:51.584 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:51 smithi102 ceph-mon[31347]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-15T12:34:51.584 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:51 smithi102 ceph-mon[31347]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-15T12:34:51.584 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:51 smithi102 ceph-mon[31347]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:52.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:52 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/4107658149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]: dispatch 2024-05-15T12:34:52.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:52 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]: dispatch 2024-05-15T12:34:52.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:52 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]': finished 2024-05-15T12:34:52.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:52 smithi173 ceph-mon[26787]: osdmap e22: 4 total, 3 up, 4 in 2024-05-15T12:34:52.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:34:52.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/4107658149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]: dispatch 2024-05-15T12:34:52.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]: dispatch 2024-05-15T12:34:52.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]': finished 2024-05-15T12:34:52.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[24799]: osdmap e22: 4 total, 3 up, 4 in 2024-05-15T12:34:52.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:34:52.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/4107658149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]: dispatch 2024-05-15T12:34:52.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]: dispatch 2024-05-15T12:34:52.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f31ac056-226b-4796-80c3-4111dad7b9fc"}]': finished 2024-05-15T12:34:52.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[31347]: osdmap e22: 4 total, 3 up, 4 in 2024-05-15T12:34:52.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:34:53.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:53 smithi173 ceph-mon[26787]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:53.901 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:53 smithi102 ceph-mon[24799]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:53.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:53 smithi102 ceph-mon[31347]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:34:54.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:54 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/1617317989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:54.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:54 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1617317989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:54.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:54 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/1617317989' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:34:55.743 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:55 smithi173 ceph-mon[26787]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-05-15T12:34:55.827 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:55 smithi102 ceph-mon[24799]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-05-15T12:34:55.827 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:55 smithi102 ceph-mon[31347]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-05-15T12:34:58.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:58 smithi173 ceph-mon[26787]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-15T12:34:58.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:58 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi102 2024-05-15T12:34:58.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:34:58 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[24799]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi102 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[31347]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.249 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:34:58.250 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi102 2024-05-15T12:34:58.250 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:34:58 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:00.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:00 smithi102 ceph-mon[24799]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-15T12:35:00.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:00 smithi102 ceph-mon[31347]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-15T12:35:00.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:00 smithi173 ceph-mon[26787]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-05-15T12:35:01.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:01 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T12:35:01.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:01 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:01.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:01 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T12:35:01.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:01 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:01.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:01 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-15T12:35:01.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:01 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:02.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:02 smithi173 ceph-mon[26787]: Deploying daemon osd.3 on smithi102 2024-05-15T12:35:02.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:02 smithi173 ceph-mon[26787]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-15T12:35:02.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:02 smithi102 ceph-mon[31347]: Deploying daemon osd.3 on smithi102 2024-05-15T12:35:02.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:02 smithi102 ceph-mon[31347]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-15T12:35:02.327 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:02 smithi102 ceph-mon[24799]: Deploying daemon osd.3 on smithi102 2024-05-15T12:35:02.328 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:02 smithi102 ceph-mon[24799]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-05-15T12:35:04.120 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:04 smithi102 ceph-mon[24799]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-05-15T12:35:04.120 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:04 smithi102 ceph-mon[31347]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-05-15T12:35:04.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:04 smithi173 ceph-mon[26787]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 74 KiB/s wr, 5 op/s 2024-05-15T12:35:06.270 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:06 smithi102 ceph-mon[24799]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-15T12:35:06.270 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:06 smithi102 ceph-mon[31347]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-15T12:35:06.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:06 smithi173 ceph-mon[26787]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-05-15T12:35:08.317 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[31347]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:08.317 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.317 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[24799]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:08.318 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:08 smithi173 ceph-mon[26787]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:08.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:08.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:08.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:08.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:08.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:10 smithi173 ceph-mon[26787]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:10.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:10 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:10 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:10 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[24799]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:10.307 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.307 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.307 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.307 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[31347]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:10.307 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.307 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.307 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:10 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:10.642 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 3 on host 'smithi102' 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.303 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:11.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:11.305 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:11 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.328 DEBUG:teuthology.orchestra.run.smithi102:osd.3> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.3.service 2024-05-15T12:35:11.331 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/nvme4n1... 2024-05-15T12:35:11.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1 2024-05-15T12:35:11.338 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:11.339 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:11 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:11.696 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:35:12.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:12 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi102 to 3760M 2024-05-15T12:35:12.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:12 smithi173 ceph-mon[26787]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:12.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:12 smithi173 ceph-mon[26787]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T12:35:12.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:12 smithi173 ceph-mon[26787]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T12:35:12.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:12.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:12 smithi173 ceph-mon[26787]: from='osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:35:12.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:12 smithi173 ceph-mon[26787]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi102 to 3760M 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[24799]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[24799]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[24799]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[24799]: from='osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[24799]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:35:12.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi102 to 3760M 2024-05-15T12:35:12.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[31347]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:12.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[31347]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-15T12:35:12.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[31347]: osdmap e23: 4 total, 3 up, 4 in 2024-05-15T12:35:12.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:12.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[31347]: from='osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:35:12.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:12 smithi102 ceph-mon[31347]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-05-15T12:35:12.403 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:35:12 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3[46488]: 2024-05-15T12:35:12.042+0000 7f87cbff9700 -1 osd.3 0 waiting for initial osdmap 2024-05-15T12:35:12.403 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:35:12 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3[46488]: 2024-05-15T12:35:12.046+0000 7f87c7dec700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:35:13.167 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:13 smithi173 ceph-mon[26787]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:35:13.167 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:13 smithi173 ceph-mon[26787]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T12:35:13.167 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:13.167 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[24799]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[24799]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[31347]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[31347]: osdmap e24: 4 total, 3 up, 4 in 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:13.696 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/nvme4n1 2024-05-15T12:35:13.696 INFO:teuthology.orchestra.run.smithi173.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:35:13.696 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-05-15T12:35:13.696 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-15T12:35:13.696 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-15T12:35:13.696 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130859 s, 801 MB/s 2024-05-15T12:35:13.696 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-15T12:35:14.242 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi173:/dev/nvme4n1 2024-05-15T12:35:14.304 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:14 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:35:14.305 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:14 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:35:14.305 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:14 smithi173 ceph-mon[26787]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:14.305 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:14 smithi173 ceph-mon[26787]: osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993] boot 2024-05-15T12:35:14.305 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:14 smithi173 ceph-mon[26787]: osdmap e25: 4 total, 4 up, 4 in 2024-05-15T12:35:14.305 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:14 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[24799]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[24799]: osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993] boot 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[24799]: osdmap e25: 4 total, 4 up, 4 in 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:35:14.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[31347]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-05-15T12:35:14.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[31347]: osd.3 [v2:172.21.15.102:6826/1700281993,v1:172.21.15.102:6827/1700281993] boot 2024-05-15T12:35:14.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[31347]: osdmap e25: 4 total, 4 up, 4 in 2024-05-15T12:35:14.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:35:14.515 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:35:15.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:15 smithi173 ceph-mon[26787]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T12:35:15.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:15 smithi102 ceph-mon[31347]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T12:35:15.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:15 smithi102 ceph-mon[24799]: osdmap e26: 4 total, 4 up, 4 in 2024-05-15T12:35:16.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:16 smithi102 ceph-mon[24799]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:16.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:16 smithi102 ceph-mon[31347]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:16.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:16 smithi173 ceph-mon[26787]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:17.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:17 smithi173 ceph-mon[26787]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:35:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:35:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:35:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[24799]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[31347]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:35:17.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:35:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:18.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:18 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi102 2024-05-15T12:35:18.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:18 smithi102 ceph-mon[24799]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:18.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:18 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi102 2024-05-15T12:35:18.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:18 smithi102 ceph-mon[31347]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:18.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:18 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi102 2024-05-15T12:35:18.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:18 smithi173 ceph-mon[26787]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/415252701' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]: dispatch 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]: dispatch 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]': finished 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[24799]: osdmap e27: 5 total, 4 up, 5 in 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/415252701' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]: dispatch 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]: dispatch 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]': finished 2024-05-15T12:35:19.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[31347]: osdmap e27: 5 total, 4 up, 5 in 2024-05-15T12:35:19.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:19 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:19.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:19 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/415252701' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]: dispatch 2024-05-15T12:35:19.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:19 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]: dispatch 2024-05-15T12:35:19.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:19 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bf0a066-545a-4f0d-8fef-a4d9136d4e69"}]': finished 2024-05-15T12:35:19.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:19 smithi173 ceph-mon[26787]: osdmap e27: 5 total, 4 up, 5 in 2024-05-15T12:35:19.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:19 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:20.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:20 smithi102 ceph-mon[31347]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:20.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:20 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/1983410369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:35:20.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:20 smithi102 ceph-mon[24799]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:20.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:20 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/1983410369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:35:20.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:20 smithi173 ceph-mon[26787]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:20.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:20 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/1983410369' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:35:22.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:22 smithi102 ceph-mon[24799]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:22.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:22 smithi102 ceph-mon[31347]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:22.416 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:22 smithi173 ceph-mon[26787]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:24.289 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:24 smithi173 ceph-mon[26787]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:24.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:24 smithi102 ceph-mon[24799]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:24.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:24 smithi102 ceph-mon[31347]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:26.311 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:26 smithi173 ceph-mon[26787]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:26.311 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:35:26.311 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:26 smithi102 ceph-mon[24799]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:35:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:26.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:26 smithi102 ceph-mon[31347]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:26.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:35:26.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:27.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:27 smithi173 ceph-mon[26787]: Deploying daemon osd.4 on smithi173 2024-05-15T12:35:27.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:27 smithi102 ceph-mon[24799]: Deploying daemon osd.4 on smithi173 2024-05-15T12:35:27.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:27 smithi102 ceph-mon[31347]: Deploying daemon osd.4 on smithi173 2024-05-15T12:35:28.374 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:28 smithi173 ceph-mon[26787]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:28.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:28 smithi102 ceph-mon[24799]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:28.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:28 smithi102 ceph-mon[31347]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:30 smithi173 ceph-mon[26787]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[24799]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[31347]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:30.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:30.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.094 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.094 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.094 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.094 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.095 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:32.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:33.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:33 smithi173 ceph-mon[26787]: Reconfiguring osd.4 (unknown last config time)... 2024-05-15T12:35:33.399 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:33 smithi173 ceph-mon[26787]: Reconfiguring daemon osd.4 on smithi173 2024-05-15T12:35:33.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:33 smithi102 ceph-mon[24799]: Reconfiguring osd.4 (unknown last config time)... 2024-05-15T12:35:33.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:33 smithi102 ceph-mon[24799]: Reconfiguring daemon osd.4 on smithi173 2024-05-15T12:35:33.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:33 smithi102 ceph-mon[31347]: Reconfiguring osd.4 (unknown last config time)... 2024-05-15T12:35:33.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:33 smithi102 ceph-mon[31347]: Reconfiguring daemon osd.4 on smithi173 2024-05-15T12:35:34.119 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:34 smithi173 ceph-mon[26787]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:34.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:34 smithi102 ceph-mon[24799]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:34 smithi102 ceph-mon[31347]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:35.088 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:35:35.088 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:35:35.088 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:35.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:35.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:35.089 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:35:35.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:35:35.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.358 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:35.359 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: from='osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.155 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.155 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-15T12:35:36.155 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:36 smithi173 ceph-mon[26787]: Cluster is now healthy 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: from='osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[31347]: Cluster is now healthy 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: osdmap e28: 5 total, 4 up, 5 in 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: from='osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:36.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-15T12:35:36.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:36 smithi102 ceph-mon[24799]: Cluster is now healthy 2024-05-15T12:35:36.487 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:37.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.068 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343] boot 2024-05-15T12:35:37.068 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: osdmap e30: 5 total, 5 up, 5 in 2024-05-15T12:35:37.068 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.094 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.4.service 2024-05-15T12:35:37.136 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/nvme3n1... 2024-05-15T12:35:37.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme3n1 2024-05-15T12:35:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:35:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T12:35:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343] boot 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: osdmap e30: 5 total, 5 up, 5 in 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: osdmap e29: 5 total, 4 up, 5 in 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: osd.4 [v2:172.21.15.173:6800/1901441343,v1:172.21.15.173:6801/1901441343] boot 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: osdmap e30: 5 total, 5 up, 5 in 2024-05-15T12:35:37.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:35:37.408 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:35:38.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:38 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:35:38.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:38 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:35:38.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:38 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi173 to 16064M 2024-05-15T12:35:38.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:38 smithi173 ceph-mon[26787]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:38.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:35:38.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:35:38.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi173 to 16064M 2024-05-15T12:35:38.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[24799]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:38.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:35:38.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:35:38.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi173 to 16064M 2024-05-15T12:35:38.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:38 smithi102 ceph-mon[31347]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 357 GiB / 358 GiB avail 2024-05-15T12:35:39.269 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/nvme3n1 2024-05-15T12:35:39.269 INFO:teuthology.orchestra.run.smithi173.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:35:39.269 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-05-15T12:35:39.270 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-15T12:35:39.270 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-15T12:35:39.270 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125601 s, 835 MB/s 2024-05-15T12:35:39.270 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-15T12:35:39.290 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:39 smithi102 ceph-mon[31347]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T12:35:39.290 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:39 smithi102 ceph-mon[24799]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T12:35:39.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:39 smithi173 ceph-mon[26787]: osdmap e31: 5 total, 5 up, 5 in 2024-05-15T12:35:39.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi173:/dev/nvme3n1 2024-05-15T12:35:40.098 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:35:40.220 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:40 smithi173 ceph-mon[26787]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:40.220 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:40 smithi173 ceph-mon[26787]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T12:35:40.291 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:40 smithi102 ceph-mon[24799]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:40.291 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:40 smithi102 ceph-mon[24799]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T12:35:40.291 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:40 smithi102 ceph-mon[31347]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:40.291 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:40 smithi102 ceph-mon[31347]: osdmap e32: 5 total, 5 up, 5 in 2024-05-15T12:35:41.127 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:41 smithi173 ceph-mon[26787]: osdmap e33: 5 total, 5 up, 5 in 2024-05-15T12:35:41.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:41 smithi102 ceph-mon[24799]: osdmap e33: 5 total, 5 up, 5 in 2024-05-15T12:35:41.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:41 smithi102 ceph-mon[31347]: osdmap e33: 5 total, 5 up, 5 in 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[24799]: pgmap v125: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[24799]: osdmap e34: 5 total, 5 up, 5 in 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[31347]: pgmap v125: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[31347]: osdmap e34: 5 total, 5 up, 5 in 2024-05-15T12:35:42.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:35:42.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:35:42.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:42 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:42.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:42 smithi173 ceph-mon[26787]: pgmap v125: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-15T12:35:42.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:42 smithi173 ceph-mon[26787]: osdmap e34: 5 total, 5 up, 5 in 2024-05-15T12:35:42.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:42 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:35:42.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:42 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:35:42.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:42 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:43.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:43 smithi102 ceph-mon[24799]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:35:43.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:43 smithi102 ceph-mon[31347]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:35:43.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:43 smithi173 ceph-mon[26787]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:35:44.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:44 smithi102 ceph-mon[24799]: pgmap v127: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-05-15T12:35:44.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:44 smithi102 ceph-mon[31347]: pgmap v127: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-05-15T12:35:44.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:44 smithi173 ceph-mon[26787]: pgmap v127: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-05-15T12:35:45.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/2922867170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]: dispatch 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]: dispatch 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]': finished 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[24799]: osdmap e35: 6 total, 5 up, 6 in 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/2922867170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]: dispatch 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]: dispatch 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]': finished 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[31347]: osdmap e35: 6 total, 5 up, 6 in 2024-05-15T12:35:45.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:35:45.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:45 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/2922867170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]: dispatch 2024-05-15T12:35:45.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:45 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]: dispatch 2024-05-15T12:35:45.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:45 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b7b81d1-9878-40a9-a805-ce2b8fcd3e83"}]': finished 2024-05-15T12:35:45.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:45 smithi173 ceph-mon[26787]: osdmap e35: 6 total, 5 up, 6 in 2024-05-15T12:35:45.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:35:46.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:46 smithi102 ceph-mon[24799]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-15T12:35:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:46 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/4114892567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:35:46.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:46 smithi102 ceph-mon[31347]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-15T12:35:46.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:46 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/4114892567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:35:46.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:46 smithi173 ceph-mon[26787]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-15T12:35:46.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:46 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/4114892567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:35:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:48 smithi102 ceph-mon[24799]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:48.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:48 smithi102 ceph-mon[31347]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:48.413 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:48 smithi173 ceph-mon[26787]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:50.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:50 smithi173 ceph-mon[26787]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:50.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:50 smithi102 ceph-mon[24799]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:50.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:50 smithi102 ceph-mon[31347]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T12:35:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T12:35:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:52 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:52.448 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:52.449 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-15T12:35:52.449 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:52 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:53.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:53 smithi173 ceph-mon[26787]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:53.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:53 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi173 2024-05-15T12:35:53.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:53 smithi173 ceph-mon[26787]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:53.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:53 smithi173 ceph-mon[26787]: Deploying daemon osd.5 on smithi173 2024-05-15T12:35:53.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:53 smithi173 ceph-mon[26787]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[24799]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi173 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[24799]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[24799]: Deploying daemon osd.5 on smithi173 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[24799]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[31347]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi173 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[31347]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[31347]: Deploying daemon osd.5 on smithi173 2024-05-15T12:35:53.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:53 smithi102 ceph-mon[31347]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-05-15T12:35:55.157 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:55.158 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.158 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.158 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.158 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:55.158 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:55.158 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:55.158 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:55.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:56 smithi173 ceph-mon[26787]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:56.116 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:56 smithi173 ceph-mon[26787]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-15T12:35:56.116 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:56 smithi173 ceph-mon[26787]: Cluster is now healthy 2024-05-15T12:35:56.116 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.116 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.116 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[24799]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[24799]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[24799]: Cluster is now healthy 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[31347]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:56.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[31347]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-05-15T12:35:56.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[31347]: Cluster is now healthy 2024-05-15T12:35:56.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:56.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:58.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:58 smithi173 ceph-mon[26787]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:58.401 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:58 smithi102 ceph-mon[24799]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:58.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:58 smithi102 ceph-mon[31347]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:59.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:59 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:59 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:59 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:59.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:59 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:59.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:59 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:59.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:59 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.439 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:35:59 smithi173 ceph-mon[26787]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:59.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[24799]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:35:59.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:35:59.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:35:59.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:35:59 smithi102 ceph-mon[31347]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:36:01.887 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:01 smithi173 ceph-mon[26787]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:36:01.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:01 smithi173 ceph-mon[26787]: from='osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:36:01.888 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:01 smithi173 ceph-mon[26787]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:36:01.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:01 smithi102 ceph-mon[24799]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:36:01.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:01 smithi102 ceph-mon[24799]: from='osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:36:01.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:01 smithi102 ceph-mon[24799]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:36:01.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:01 smithi102 ceph-mon[31347]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:36:01.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:01 smithi102 ceph-mon[31347]: from='osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:36:01.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:01 smithi102 ceph-mon[31347]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-15T12:36:01.953 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-05-15T12:36:02.569 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.5.service 2024-05-15T12:36:02.572 INFO:tasks.cephadm:Deploying osd.6 on smithi173 with /dev/nvme2n1... 2024-05-15T12:36:02.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme2n1 2024-05-15T12:36:02.852 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: osdmap e36: 6 total, 5 up, 6 in 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:02.976 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi173 to 8032M 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: osdmap e37: 6 total, 5 up, 6 in 2024-05-15T12:36:02.977 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:02 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:03.113 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T12:36:03.113 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:03.113 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: osdmap e36: 6 total, 5 up, 6 in 2024-05-15T12:36:03.113 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:03.113 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi173 to 8032M 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: osdmap e37: 6 total, 5 up, 6 in 2024-05-15T12:36:03.114 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: osdmap e36: 6 total, 5 up, 6 in 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:03.115 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi173 to 8032M 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: osdmap e37: 6 total, 5 up, 6 in 2024-05-15T12:36:03.116 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:02 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:03.244 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:36:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:36:03.172+0000 7f220dc65700 -1 osd.5 0 waiting for initial osdmap 2024-05-15T12:36:03.244 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:36:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:36:03.177+0000 7f2205a50700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:36:03.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:03 smithi173 ceph-mon[26787]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:36:03.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:03.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:03 smithi173 ceph-mon[26787]: from='osd.5 ' entity='osd.5' 2024-05-15T12:36:03.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:03 smithi173 ceph-mon[26787]: osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521] boot 2024-05-15T12:36:03.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:03 smithi173 ceph-mon[26787]: osdmap e38: 6 total, 6 up, 6 in 2024-05-15T12:36:03.980 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:03 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[24799]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[24799]: from='osd.5 ' entity='osd.5' 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[24799]: osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521] boot 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[24799]: osdmap e38: 6 total, 6 up, 6 in 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[31347]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-05-15T12:36:04.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:04.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[31347]: from='osd.5 ' entity='osd.5' 2024-05-15T12:36:04.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[31347]: osd.5 [v2:172.21.15.173:6808/2038365521,v1:172.21.15.173:6809/2038365521] boot 2024-05-15T12:36:04.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[31347]: osdmap e38: 6 total, 6 up, 6 in 2024-05-15T12:36:04.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:03 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:36:04.771 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/nvme2n1 2024-05-15T12:36:04.771 INFO:teuthology.orchestra.run.smithi173.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:36:04.772 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-05-15T12:36:04.772 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-15T12:36:04.772 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-15T12:36:04.772 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130862 s, 801 MB/s 2024-05-15T12:36:04.772 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-15T12:36:04.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:04 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:36:04.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:04 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:36:04.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:04 smithi173 ceph-mon[26787]: osdmap e39: 6 total, 6 up, 6 in 2024-05-15T12:36:05.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:04 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:36:05.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:04 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:36:05.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:04 smithi102 ceph-mon[24799]: osdmap e39: 6 total, 6 up, 6 in 2024-05-15T12:36:05.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:04 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:36:05.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:04 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:36:05.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:04 smithi102 ceph-mon[31347]: osdmap e39: 6 total, 6 up, 6 in 2024-05-15T12:36:05.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi173:/dev/nvme2n1 2024-05-15T12:36:05.577 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:36:05.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:05 smithi173 ceph-mon[26787]: pgmap v144: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:05.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:05 smithi173 ceph-mon[26787]: osdmap e40: 6 total, 6 up, 6 in 2024-05-15T12:36:06.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:05 smithi102 ceph-mon[31347]: pgmap v144: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:06.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:05 smithi102 ceph-mon[31347]: osdmap e40: 6 total, 6 up, 6 in 2024-05-15T12:36:06.053 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:05 smithi102 ceph-mon[24799]: pgmap v144: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:06.054 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:05 smithi102 ceph-mon[24799]: osdmap e40: 6 total, 6 up, 6 in 2024-05-15T12:36:07.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:07 smithi173 ceph-mon[26787]: pgmap v147: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:07.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:07 smithi173 ceph-mon[26787]: osdmap e41: 6 total, 6 up, 6 in 2024-05-15T12:36:07.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:07 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:36:07.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:07 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:36:07.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:07 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:08.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[24799]: pgmap v147: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:08.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[24799]: osdmap e41: 6 total, 6 up, 6 in 2024-05-15T12:36:08.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:36:08.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:36:08.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:08.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[31347]: pgmap v147: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:08.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[31347]: osdmap e41: 6 total, 6 up, 6 in 2024-05-15T12:36:08.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:36:08.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:36:08.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:07 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:08.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:08 smithi173 ceph-mon[26787]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:36:08.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:08 smithi173 ceph-mon[26787]: osdmap e42: 6 total, 6 up, 6 in 2024-05-15T12:36:09.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:08 smithi102 ceph-mon[24799]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:36:09.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:08 smithi102 ceph-mon[24799]: osdmap e42: 6 total, 6 up, 6 in 2024-05-15T12:36:09.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:08 smithi102 ceph-mon[31347]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:36:09.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:08 smithi102 ceph-mon[31347]: osdmap e42: 6 total, 6 up, 6 in 2024-05-15T12:36:09.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:09 smithi173 ceph-mon[26787]: pgmap v150: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:10.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:09 smithi102 ceph-mon[24799]: pgmap v150: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:10.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:09 smithi102 ceph-mon[31347]: pgmap v150: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:11.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/3928594559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]: dispatch 2024-05-15T12:36:11.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]: dispatch 2024-05-15T12:36:11.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]': finished 2024-05-15T12:36:11.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[24799]: osdmap e43: 7 total, 6 up, 7 in 2024-05-15T12:36:11.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:11.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/3928594559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]: dispatch 2024-05-15T12:36:11.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]: dispatch 2024-05-15T12:36:11.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]': finished 2024-05-15T12:36:11.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[31347]: osdmap e43: 7 total, 6 up, 7 in 2024-05-15T12:36:11.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:10 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:11.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:10 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/3928594559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]: dispatch 2024-05-15T12:36:11.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:10 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]: dispatch 2024-05-15T12:36:11.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:10 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c5cada8-7d3c-40d7-b215-33d221ad63ca"}]': finished 2024-05-15T12:36:11.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:10 smithi173 ceph-mon[26787]: osdmap e43: 7 total, 6 up, 7 in 2024-05-15T12:36:11.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:10 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:12.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:11 smithi102 ceph-mon[24799]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-05-15T12:36:12.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:11 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/164860063' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:36:12.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:11 smithi102 ceph-mon[31347]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-05-15T12:36:12.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:11 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/164860063' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:36:12.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:11 smithi173 ceph-mon[26787]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-05-15T12:36:12.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:11 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/164860063' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:36:14.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:13 smithi102 ceph-mon[31347]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-15T12:36:14.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:13 smithi102 ceph-mon[24799]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-15T12:36:14.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:13 smithi173 ceph-mon[26787]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-15T12:36:15.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi173 2024-05-15T12:36:15.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[24799]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-15T12:36:15.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi173 2024-05-15T12:36:15.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:15 smithi102 ceph-mon[31347]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-15T12:36:15.991 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.991 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.991 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:15 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi173 2024-05-15T12:36:15.991 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:15 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:15.991 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:15 smithi173 ceph-mon[26787]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-05-15T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.120 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.151 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:17.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:18.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:17 smithi102 ceph-mon[24799]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:36:18.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:17 smithi102 ceph-mon[31347]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:36:18.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:17 smithi173 ceph-mon[26787]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-05-15T12:36:18.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:18 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T12:36:18.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:18 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:19.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:18 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T12:36:19.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:18 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:19.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:18 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-15T12:36:19.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:18 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:19.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:19 smithi102 ceph-mon[24799]: Deploying daemon osd.6 on smithi173 2024-05-15T12:36:19.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:19 smithi102 ceph-mon[24799]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-15T12:36:19.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:19 smithi102 ceph-mon[31347]: Deploying daemon osd.6 on smithi173 2024-05-15T12:36:19.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:19 smithi102 ceph-mon[31347]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-15T12:36:19.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:19 smithi173 ceph-mon[26787]: Deploying daemon osd.6 on smithi173 2024-05-15T12:36:19.993 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:19 smithi173 ceph-mon[26787]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-15T12:36:21.934 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:21 smithi173 ceph-mon[26787]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:22.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:21 smithi102 ceph-mon[24799]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:22.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:21 smithi102 ceph-mon[31347]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:24.151 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:23 smithi173 ceph-mon[26787]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:24.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:23 smithi102 ceph-mon[24799]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:24.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:23 smithi102 ceph-mon[31347]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:26 smithi173 ceph-mon[26787]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:26 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[24799]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[31347]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:26.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:26 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:28 smithi173 ceph-mon[26787]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:28.348 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 6 on host 'smithi173' 2024-05-15T12:36:28.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:28 smithi102 ceph-mon[24799]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:28.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:28 smithi102 ceph-mon[31347]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:28.946 DEBUG:teuthology.orchestra.run.smithi173:osd.6> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.6.service 2024-05-15T12:36:28.949 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/nvme1n1... 2024-05-15T12:36:28.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/nvme1n1 2024-05-15T12:36:29.226 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:36:29.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:36:29.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:36:29.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:36:29.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:36:29.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:29.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:29.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:29.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:29.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:29.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:29.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:30.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:30 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi173 to 5354M 2024-05-15T12:36:30.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:30 smithi173 ceph-mon[26787]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:30.350 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:30 smithi173 ceph-mon[26787]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T12:36:30.350 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:30 smithi173 ceph-mon[26787]: from='osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:30.350 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:30 smithi173 ceph-mon[26787]: osdmap e44: 7 total, 6 up, 7 in 2024-05-15T12:36:30.350 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:30.350 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:30 smithi173 ceph-mon[26787]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi173 to 5354M 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[24799]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[24799]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[24799]: from='osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[24799]: osdmap e44: 7 total, 6 up, 7 in 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[24799]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:30.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi173 to 5354M 2024-05-15T12:36:30.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[31347]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:30.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[31347]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-15T12:36:30.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[31347]: from='osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:30.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[31347]: osdmap e44: 7 total, 6 up, 7 in 2024-05-15T12:36:30.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:30.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:30 smithi102 ceph-mon[31347]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:30.676 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:36:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:36:30.488+0000 7fb797c09700 -1 osd.6 0 waiting for initial osdmap 2024-05-15T12:36:30.676 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:36:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:36:30.498+0000 7fb78f9f4700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:36:31.059 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/nvme1n1 2024-05-15T12:36:31.059 INFO:teuthology.orchestra.run.smithi173.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-05-15T12:36:31.059 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-05-15T12:36:31.059 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-05-15T12:36:31.059 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-05-15T12:36:31.059 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135028 s, 777 MB/s 2024-05-15T12:36:31.060 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-05-15T12:36:31.362 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:31 smithi173 ceph-mon[26787]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:31.362 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:31 smithi173 ceph-mon[26787]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T12:36:31.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:31.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:31 smithi173 ceph-mon[26787]: from='osd.6 ' entity='osd.6' 2024-05-15T12:36:31.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:31 smithi173 ceph-mon[26787]: osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336] boot 2024-05-15T12:36:31.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:31 smithi173 ceph-mon[26787]: osdmap e46: 7 total, 7 up, 7 in 2024-05-15T12:36:31.363 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:31.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[31347]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:31.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[31347]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T12:36:31.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:31.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[31347]: from='osd.6 ' entity='osd.6' 2024-05-15T12:36:31.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[31347]: osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336] boot 2024-05-15T12:36:31.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[31347]: osdmap e46: 7 total, 7 up, 7 in 2024-05-15T12:36:31.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[24799]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[24799]: osdmap e45: 7 total, 6 up, 7 in 2024-05-15T12:36:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[24799]: from='osd.6 ' entity='osd.6' 2024-05-15T12:36:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[24799]: osd.6 [v2:172.21.15.173:6816/2130176336,v1:172.21.15.173:6817/2130176336] boot 2024-05-15T12:36:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[24799]: osdmap e46: 7 total, 7 up, 7 in 2024-05-15T12:36:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:36:31.694 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi173:/dev/nvme1n1 2024-05-15T12:36:31.971 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:36:32.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:32 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:36:32.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:32 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:36:32.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:32 smithi173 ceph-mon[26787]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:32.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:32 smithi173 ceph-mon[26787]: osdmap e47: 7 total, 7 up, 7 in 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[31347]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[31347]: osdmap e47: 7 total, 7 up, 7 in 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[24799]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-05-15T12:36:32.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:32 smithi102 ceph-mon[24799]: osdmap e47: 7 total, 7 up, 7 in 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[31347]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[31347]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[24799]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[24799]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T12:36:34.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:36:34.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:36:34.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:34.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:33 smithi173 ceph-mon[26787]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:34.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:33 smithi173 ceph-mon[26787]: osdmap e48: 7 total, 7 up, 7 in 2024-05-15T12:36:34.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:33 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-15T12:36:34.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:33 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-15T12:36:34.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:33 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:35.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:36:35.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: osdmap e49: 7 total, 7 up, 7 in 2024-05-15T12:36:35.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:35.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:35.219 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:35 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: osdmap e49: 7 total, 7 up, 7 in 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:35.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: osdmap e49: 7 total, 7 up, 7 in 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:35.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:35 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.192 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi173 2024-05-15T12:36:36.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: osdmap e50: 7 total, 7 up, 7 in 2024-05-15T12:36:36.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:36 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi173 2024-05-15T12:36:36.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.303 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: osdmap e50: 7 total, 7 up, 7 in 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi173 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: osdmap e50: 7 total, 7 up, 7 in 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:36.304 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:36 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:37.319 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/1294322661' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]: dispatch 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]: dispatch 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[31347]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]': finished 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[31347]: osdmap e51: 8 total, 7 up, 8 in 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/1294322661' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]: dispatch 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]: dispatch 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[24799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]': finished 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[24799]: osdmap e51: 8 total, 7 up, 8 in 2024-05-15T12:36:37.320 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:37 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:37 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/1294322661' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]: dispatch 2024-05-15T12:36:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:37 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]: dispatch 2024-05-15T12:36:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:37 smithi173 ceph-mon[26787]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fd4b9b4c-13f2-49ac-8106-bcbc89a7b765"}]': finished 2024-05-15T12:36:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:37 smithi173 ceph-mon[26787]: osdmap e51: 8 total, 7 up, 8 in 2024-05-15T12:36:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:37 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:38.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:38 smithi102 ceph-mon[24799]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 199 KiB/s, 0 objects/s recovering 2024-05-15T12:36:38.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:38 smithi102 ceph-mon[31347]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 199 KiB/s, 0 objects/s recovering 2024-05-15T12:36:38.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:38 smithi173 ceph-mon[26787]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 199 KiB/s, 0 objects/s recovering 2024-05-15T12:36:39.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:39 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/3364272912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:36:39.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:39 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/3364272912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:36:39.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:39 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/3364272912' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-15T12:36:40.378 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:40 smithi173 ceph-mon[26787]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-15T12:36:40.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:40 smithi102 ceph-mon[24799]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-15T12:36:40.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:40 smithi102 ceph-mon[31347]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-05-15T12:36:42.307 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:42 smithi173 ceph-mon[26787]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-15T12:36:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:42 smithi102 ceph-mon[24799]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-15T12:36:42.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:42 smithi102 ceph-mon[31347]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-05-15T12:36:44.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:44 smithi173 ceph-mon[26787]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-15T12:36:44.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:44 smithi102 ceph-mon[24799]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-15T12:36:44.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:44 smithi102 ceph-mon[31347]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-15T12:36:45.336 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T12:36:45.336 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:45.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T12:36:45.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:45.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-15T12:36:45.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:46.347 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:46 smithi173 ceph-mon[26787]: Deploying daemon osd.7 on smithi173 2024-05-15T12:36:46.347 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:46 smithi173 ceph-mon[26787]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-15T12:36:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:46 smithi102 ceph-mon[24799]: Deploying daemon osd.7 on smithi173 2024-05-15T12:36:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:46 smithi102 ceph-mon[24799]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-15T12:36:46.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:46 smithi102 ceph-mon[31347]: Deploying daemon osd.7 on smithi173 2024-05-15T12:36:46.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:46 smithi102 ceph-mon[31347]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-15T12:36:48.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:48 smithi173 ceph-mon[26787]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:48.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:48 smithi102 ceph-mon[24799]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:48.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:48 smithi102 ceph-mon[31347]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:50.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:50 smithi102 ceph-mon[24799]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:50.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:50 smithi102 ceph-mon[31347]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:50 smithi173 ceph-mon[26787]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:51.460 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.460 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:51.460 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:51.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:51.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:51 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:51 smithi173 ceph-mon[26787]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:51.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:51.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:51.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:51.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[31347]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:51.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:51.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:51.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:51.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:51.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:51 smithi102 ceph-mon[24799]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:53.395 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:53.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:53.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:53.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:53.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:53.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:53.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:53 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:53.998 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.7.service 2024-05-15T12:36:54.001 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-05-15T12:36:54.002 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-15T12:36:54.259 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.377 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-15T12:36:54.378 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:36:54.379 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:53 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: Adjusting osd_memory_target on smithi173 to 4016M 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: osdmap e52: 8 total, 7 up, 8 in 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: from='osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:55 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.245 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:36:55 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:36:55.005+0000 7f5991a74700 -1 osd.7 0 waiting for initial osdmap 2024-05-15T12:36:55.245 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:36:55 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:36:55.013+0000 7f598d867700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: Adjusting osd_memory_target on smithi173 to 4016M 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: osdmap e52: 8 total, 7 up, 8 in 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: from='osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: Adjusting osd_memory_target on smithi173 to 4016M 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: osdmap e52: 8 total, 7 up, 8 in 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: from='osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:55.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:55 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:56.220 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:36:56.220 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi173 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: osdmap e53: 8 total, 7 up, 8 in 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878] boot 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: osdmap e54: 8 total, 8 up, 8 in 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi173 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:56.221 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:56.222 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: osdmap e53: 8 total, 7 up, 8 in 2024-05-15T12:36:56.222 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.222 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.222 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878] boot 2024-05-15T12:36:56.222 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: osdmap e54: 8 total, 8 up, 8 in 2024-05-15T12:36:56.222 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:56 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi173 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: osdmap e53: 8 total, 7 up, 8 in 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: osd.7 [v2:172.21.15.173:6824/467039878,v1:172.21.15.173:6825/467039878] boot 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: osdmap e54: 8 total, 8 up, 8 in 2024-05-15T12:36:56.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:56 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:36:56.862 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1715776615,"num_in_osds":8,"osd_in_since":1715776596,"num_remapped_pgs":1} 2024-05-15T12:36:56.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T12:36:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:36:57.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: purged_snaps scrub starts 2024-05-15T12:36:57.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: purged_snaps scrub ok 2024-05-15T12:36:57.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/54365645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T12:36:57.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: osdmap e55: 8 total, 8 up, 8 in 2024-05-15T12:36:57.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:57 smithi173 ceph-mon[26787]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: purged_snaps scrub starts 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: purged_snaps scrub ok 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/54365645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: osdmap e55: 8 total, 8 up, 8 in 2024-05-15T12:36:57.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[24799]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: purged_snaps scrub starts 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: purged_snaps scrub ok 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/54365645' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: osdmap e55: 8 total, 8 up, 8 in 2024-05-15T12:36:57.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:57 smithi102 ceph-mon[31347]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:36:58.980 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:36:58.980 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":57,"fsid":"eac36fa0-12b6-11ef-bc98-c7b262605968","created":"2024-05-15T12:31:20.823409+0000","modified":"2024-05-15T12:36:58.849511+0000","last_up_change":"2024-05-15T12:36:55.840485+0000","last_in_change":"2024-05-15T12:36:36.892145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:34:45.237868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a157badc-d604-423e-80fd-7205437c36de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6803","nonce":4160180593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6805","nonce":4160180593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6809","nonce":4160180593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6807","nonce":4160180593}]},"public_addr":"172.21.15.102:6803/4160180593","cluster_addr":"172.21.15.102:6805/4160180593","heartbeat_back_addr":"172.21.15.102:6809/4160180593","heartbeat_front_addr":"172.21.15.102:6807/4160180593","state":["exists","up"]},{"osd":1,"uuid":"3cfe8e1f-978d-4e11-94f3-094a13ac15fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6811","nonce":838755088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6813","nonce":838755088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6817","nonce":838755088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6815","nonce":838755088}]},"public_addr":"172.21.15.102:6811/838755088","cluster_addr":"172.21.15.102:6813/838755088","heartbeat_back_addr":"172.21.15.102:6817/838755088","heartbeat_front_addr":"172.21.15.102:6815/838755088","state":["exists","up"]},{"osd":2,"uuid":"f49f2eb3-557b-456c-91ab-7098fd58995a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6819","nonce":1924240300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6821","nonce":1924240300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6825","nonce":1924240300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6823","nonce":1924240300}]},"public_addr":"172.21.15.102:6819/1924240300","cluster_addr":"172.21.15.102:6821/1924240300","heartbeat_back_addr":"172.21.15.102:6825/1924240300","heartbeat_front_addr":"172.21.15.102:6823/1924240300","state":["exists","up"]},{"osd":3,"uuid":"f31ac056-226b-4796-80c3-4111dad7b9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6827","nonce":1700281993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6829","nonce":1700281993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6833","nonce":1700281993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6831","nonce":1700281993}]},"public_addr":"172.21.15.102:6827/1700281993","cluster_addr":"172.21.15.102:6829/1700281993","heartbeat_back_addr":"172.21.15.102:6833/1700281993","heartbeat_front_addr":"172.21.15.102:6831/1700281993","state":["exists","up"]},{"osd":4,"uuid":"1bf0a066-545a-4f0d-8fef-a4d9136d4e69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6801","nonce":1901441343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6803","nonce":1901441343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6807","nonce":1901441343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6805","nonce":1901441343}]},"public_addr":"172.21.15.173:6801/1901441343","cluster_addr":"172.21.15.173:6803/1901441343","heartbeat_back_addr":"172.21.15.173:6807/1901441343","heartbeat_front_addr":"172.21.15.173:6805/1901441343","state":["exists","up"]},{"osd":5,"uuid":"1b7b81d1-9878-40a9-a805-ce2b8fcd3e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6809","nonce":2038365521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6811","nonce":2038365521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6815","nonce":2038365521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6813","nonce":2038365521}]},"public_addr":"172.21.15.173:6809/2038365521","cluster_addr":"172.21.15.173:6811/2038365521","heartbeat_back_addr":"172.21.15.173:6815/2038365521","heartbeat_front_addr":"172.21.15.173:6813/2038365521","state":["exists","up"]},{"osd":6,"uuid":"5c5cada8-7d3c-40d7-b215-33d221ad63ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6817","nonce":2130176336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6819","nonce":2130176336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6823","nonce":2130176336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6821","nonce":2130176336}]},"public_addr":"172.21.15.173:6817/2130176336","cluster_addr":"172.21.15.173:6819/2130176336","heartbeat_back_addr":"172.21.15.173:6823/2130176336","heartbeat_front_addr":"172.21.15.173:6821/2130176336","state":["exists","up"]},{"osd":7,"uuid":"fd4b9b4c-13f2-49ac-8106-bcbc89a7b765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6825","nonce":467039878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6827","nonce":467039878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6831","nonce":467039878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6829","nonce":467039878}]},"public_addr":"172.21.15.173:6825/467039878","cluster_addr":"172.21.15.173:6827/467039878","heartbeat_back_addr":"172.21.15.173:6831/467039878","heartbeat_front_addr":"172.21.15.173:6829/467039878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:33:51.093575+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:34:16.553531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:34:42.922376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:35:11.581849+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:35:35.862174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:02.347318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:29.113689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:54.372914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2982719131":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/785097316":"2024-05-16T12:32:22.919753+0000","172.21.15.102:6801/3724371257":"2024-05-16T12:32:22.919753+0000","172.21.15.102:6800/3724371257":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/4275670978":"2024-05-16T12:31:56.860756+0000","172.21.15.102:0/2007662139":"2024-05-16T12:31:56.860756+0000","172.21.15.102:6800/455878624":"2024-05-16T12:31:56.860756+0000","172.21.15.102:6801/455878624":"2024-05-16T12:31:56.860756+0000","172.21.15.102:0/2850653973":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/2344980692":"2024-05-16T12:31:56.860756+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:36:59.112 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:58 smithi173 ceph-mon[26787]: osdmap e56: 8 total, 8 up, 8 in 2024-05-15T12:36:59.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:58 smithi102 ceph-mon[24799]: osdmap e56: 8 total, 8 up, 8 in 2024-05-15T12:36:59.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:58 smithi102 ceph-mon[31347]: osdmap e56: 8 total, 8 up, 8 in 2024-05-15T12:36:59.651 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-15T12:34:45.237868+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-15T12:36:59.652 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-15T12:36:59.891 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:36:59.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:59 smithi102 ceph-mon[31347]: osdmap e57: 8 total, 8 up, 8 in 2024-05-15T12:36:59.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:59 smithi102 ceph-mon[31347]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:36:59.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:36:59 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2359744358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:36:59.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:59 smithi102 ceph-mon[24799]: osdmap e57: 8 total, 8 up, 8 in 2024-05-15T12:36:59.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:59 smithi102 ceph-mon[24799]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:36:59.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:36:59 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2359744358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:36:59.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:59 smithi173 ceph-mon[26787]: osdmap e57: 8 total, 8 up, 8 in 2024-05-15T12:36:59.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:59 smithi173 ceph-mon[26787]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:36:59.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:36:59 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2359744358' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:37:01.049 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:00 smithi102 ceph-mon[24799]: osdmap e58: 8 total, 8 up, 8 in 2024-05-15T12:37:01.049 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:00 smithi102 ceph-mon[31347]: osdmap e58: 8 total, 8 up, 8 in 2024-05-15T12:37:01.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:00 smithi173 ceph-mon[26787]: osdmap e58: 8 total, 8 up, 8 in 2024-05-15T12:37:01.736 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-05-15T12:37:01.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:01 smithi102 ceph-mon[31347]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:01.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:01 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/3092790177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T12:37:01.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:01 smithi102 ceph-mon[24799]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:01.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:01 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3092790177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T12:37:02.145 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:01 smithi173 ceph-mon[26787]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:02.145 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:01 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/3092790177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-15T12:37:02.451 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi102 2024-05-15T12:37:02.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi102=foo.a' 2024-05-15T12:37:02.704 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:04.086 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:03 smithi173 ceph-mon[26787]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-15T12:37:04.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:03 smithi102 ceph-mon[24799]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-15T12:37:04.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:03 smithi102 ceph-mon[31347]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-15T12:37:04.518 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled rgw.foo.a update... 2024-05-15T12:37:05.090 DEBUG:teuthology.orchestra.run.smithi102:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@rgw.foo.a.service 2024-05-15T12:37:05.093 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi173 2024-05-15T12:37:05.094 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-05-15T12:37:05.328 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-15T12:37:05.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:05.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: Deploying daemon rgw.foo.a on smithi102 2024-05-15T12:37:05.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:05 smithi173 ceph-mon[26787]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-05-15T12:37:05.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:05.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-05-15T12:37:05.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: Deploying daemon rgw.foo.a on smithi102 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[24799]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-05-15T12:37:05.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi102=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: Deploying daemon rgw.foo.a on smithi102 2024-05-15T12:37:05.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:05 smithi102 ceph-mon[31347]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-05-15T12:37:06.902 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:37:06 smithi102 systemd[1]: Starting Ceph rgw.foo.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:37:07.285 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:37:07 smithi102 podman[50120]: 2024-05-15 12:37:07.001171379 +0000 UTC m=+0.110563661 container create cd31bb0d8a84cc8297a52c989fae94b4f22a8601e640fac3bac9c07d6ade8f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-15T12:37:07.285 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:37:07 smithi102 podman[50120]: 2024-05-15 12:37:06.908323011 +0000 UTC m=+0.017715290 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:37:07.285 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:37:07 smithi102 podman[50120]: 2024-05-15 12:37:07.221182983 +0000 UTC m=+0.330575271 container init cd31bb0d8a84cc8297a52c989fae94b4f22a8601e640fac3bac9c07d6ade8f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:37:07.285 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:37:07 smithi102 podman[50120]: 2024-05-15 12:37:07.224570067 +0000 UTC m=+0.333962340 container start cd31bb0d8a84cc8297a52c989fae94b4f22a8601e640fac3bac9c07d6ade8f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514) 2024-05-15T12:37:07.285 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:37:07 smithi102 bash[50120]: cd31bb0d8a84cc8297a52c989fae94b4f22a8601e640fac3bac9c07d6ade8f03 2024-05-15T12:37:07.652 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:37:07 smithi102 systemd[1]: Started Ceph rgw.foo.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:37:08.049 INFO:teuthology.orchestra.run.smithi173.stderr:pool 'datapool' created 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/618398790' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.300 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/618398790' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.301 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.302 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:08.302 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:08.302 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:08 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/618398790' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:08.379 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:08 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:08.659 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- rbd pool init datapool 2024-05-15T12:37:08.900 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:09.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-05-15T12:37:09.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: Checking dashboard <-> RGW credentials 2024-05-15T12:37:09.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-05-15T12:37:09.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:09.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: osdmap e59: 8 total, 8 up, 8 in 2024-05-15T12:37:09.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2160679456' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.305 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2903543624' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: Checking dashboard <-> RGW credentials 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: osdmap e59: 8 total, 8 up, 8 in 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2160679456' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2903543624' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.306 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:09 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: Saving service rgw.foo.a spec with placement smithi102=foo.a;count:1 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: Checking dashboard <-> RGW credentials 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: osdmap e59: 8 total, 8 up, 8 in 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2160679456' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2903543624' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:09.340 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:09 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-05-15T12:37:10.330 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:10 smithi173 ceph-mon[26787]: pgmap v199: 36 pgs: 6 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-15T12:37:10.330 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:10 smithi173 ceph-mon[26787]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:37:10.330 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:10 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-15T12:37:10.330 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:10 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-15T12:37:10.330 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:10 smithi173 ceph-mon[26787]: osdmap e60: 8 total, 8 up, 8 in 2024-05-15T12:37:10.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[24799]: pgmap v199: 36 pgs: 6 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-15T12:37:10.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[24799]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:37:10.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-15T12:37:10.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-15T12:37:10.346 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[24799]: osdmap e60: 8 total, 8 up, 8 in 2024-05-15T12:37:10.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[31347]: pgmap v199: 36 pgs: 6 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-05-15T12:37:10.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[31347]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:37:10.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-15T12:37:10.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-05-15T12:37:10.347 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:10 smithi102 ceph-mon[31347]: osdmap e60: 8 total, 8 up, 8 in 2024-05-15T12:37:11.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[24799]: osdmap e61: 8 total, 8 up, 8 in 2024-05-15T12:37:11.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/4207126241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[24799]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[31347]: osdmap e61: 8 total, 8 up, 8 in 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/4207126241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-05-15T12:37:11.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:11 smithi102 ceph-mon[31347]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-05-15T12:37:11.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:11 smithi173 ceph-mon[26787]: osdmap e61: 8 total, 8 up, 8 in 2024-05-15T12:37:11.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:11 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:11 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:11 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:11 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-05-15T12:37:11.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:11 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/4207126241' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-05-15T12:37:11.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:11 smithi173 ceph-mon[26787]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-05-15T12:37:12.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[31347]: pgmap v202: 68 pgs: 13 creating+peering, 11 active+clean, 44 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-05-15T12:37:12.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-05-15T12:37:12.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-05-15T12:37:12.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[31347]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-05-15T12:37:12.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[31347]: osdmap e62: 8 total, 8 up, 8 in 2024-05-15T12:37:12.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:12.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[24799]: pgmap v202: 68 pgs: 13 creating+peering, 11 active+clean, 44 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-05-15T12:37:12.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-05-15T12:37:12.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-05-15T12:37:12.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[24799]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-05-15T12:37:12.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[24799]: osdmap e62: 8 total, 8 up, 8 in 2024-05-15T12:37:12.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:12 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:12.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:12 smithi173 ceph-mon[26787]: pgmap v202: 68 pgs: 13 creating+peering, 11 active+clean, 44 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-05-15T12:37:12.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:12 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-05-15T12:37:12.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:12 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-05-15T12:37:12.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:12 smithi173 ceph-mon[26787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-05-15T12:37:12.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:12 smithi173 ceph-mon[26787]: osdmap e62: 8 total, 8 up, 8 in 2024-05-15T12:37:12.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:12 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: osdmap e63: 8 total, 8 up, 8 in 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:13 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: osdmap e63: 8 total, 8 up, 8 in 2024-05-15T12:37:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: osdmap e63: 8 total, 8 up, 8 in 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.404 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:13 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:13.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.173 --placement '1;smithi173=iscsi.a' 2024-05-15T12:37:14.042 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:14.377 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:14 smithi173 ceph-mon[26787]: pgmap v205: 100 pgs: 16 creating+peering, 33 active+clean, 51 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-15T12:37:14.377 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:14 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:14.377 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:14 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-05-15T12:37:14.377 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:14 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-05-15T12:37:14.377 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:14 smithi173 ceph-mon[26787]: osdmap e64: 8 total, 8 up, 8 in 2024-05-15T12:37:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[24799]: pgmap v205: 100 pgs: 16 creating+peering, 33 active+clean, 51 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-15T12:37:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-05-15T12:37:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-05-15T12:37:14.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[24799]: osdmap e64: 8 total, 8 up, 8 in 2024-05-15T12:37:14.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[31347]: pgmap v205: 100 pgs: 16 creating+peering, 33 active+clean, 51 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-15T12:37:14.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:14.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-05-15T12:37:14.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-05-15T12:37:14.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:14 smithi102 ceph-mon[31347]: osdmap e64: 8 total, 8 up, 8 in 2024-05-15T12:37:15.343 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:15 smithi173 ceph-mon[26787]: osdmap e65: 8 total, 8 up, 8 in 2024-05-15T12:37:15.343 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:15 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.343 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:15 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.343 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:15 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.343 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:15 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[24799]: osdmap e65: 8 total, 8 up, 8 in 2024-05-15T12:37:15.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[31347]: osdmap e65: 8 total, 8 up, 8 in 2024-05-15T12:37:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:15 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-05-15T12:37:15.801 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled iscsi.datapool update... 2024-05-15T12:37:16.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: pgmap v208: 132 pgs: 12 creating+peering, 80 active+clean, 40 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-15T12:37:16.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-05-15T12:37:16.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-05-15T12:37:16.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: osdmap e66: 8 total, 8 up, 8 in 2024-05-15T12:37:16.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[24799]: osdmap e67: 8 total, 8 up, 8 in 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: pgmap v208: 132 pgs: 12 creating+peering, 80 active+clean, 40 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-05-15T12:37:16.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: osdmap e66: 8 total, 8 up, 8 in 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-15T12:37:16.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:16 smithi102 ceph-mon[31347]: osdmap e67: 8 total, 8 up, 8 in 2024-05-15T12:37:16.413 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: pgmap v208: 132 pgs: 12 creating+peering, 80 active+clean, 40 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: osdmap e66: 8 total, 8 up, 8 in 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2162785748' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/190528192' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-05-15T12:37:16.414 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:16 smithi173 ceph-mon[26787]: osdmap e67: 8 total, 8 up, 8 in 2024-05-15T12:37:16.709 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-05-15T12:37:16.709 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:37:16.709 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-05-15T12:37:16.739 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:37:16.739 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-05-15T12:37:16.767 DEBUG:teuthology.orchestra.run.smithi173:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@iscsi.iscsi.a.service 2024-05-15T12:37:16.810 INFO:tasks.cephadm:Adding prometheus.a on smithi173 2024-05-15T12:37:16.810 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply prometheus '1;smithi173=a' 2024-05-15T12:37:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-05-15T12:37:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-15T12:37:17.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-15T12:37:17.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:17 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:17.424 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:17.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:17.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-05-15T12:37:17.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:17.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-05-15T12:37:17.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-05-15T12:37:17.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:17 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:18.369 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[31347]: Deploying daemon iscsi.iscsi.a on smithi173 2024-05-15T12:37:18.369 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[31347]: pgmap v211: 132 pgs: 8 creating+peering, 104 active+clean, 20 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-15T12:37:18.369 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[31347]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:37:18.369 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[31347]: Cluster is now healthy 2024-05-15T12:37:18.370 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[24799]: Deploying daemon iscsi.iscsi.a on smithi173 2024-05-15T12:37:18.370 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[24799]: pgmap v211: 132 pgs: 8 creating+peering, 104 active+clean, 20 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-15T12:37:18.370 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[24799]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:37:18.370 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:18 smithi102 ceph-mon[24799]: Cluster is now healthy 2024-05-15T12:37:18.390 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:18 smithi173 ceph-mon[26787]: Deploying daemon iscsi.iscsi.a on smithi173 2024-05-15T12:37:18.390 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:18 smithi173 ceph-mon[26787]: pgmap v211: 132 pgs: 8 creating+peering, 104 active+clean, 20 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-15T12:37:18.390 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:18 smithi173 ceph-mon[26787]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-15T12:37:18.390 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:18 smithi173 ceph-mon[26787]: Cluster is now healthy 2024-05-15T12:37:20.177 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:20 smithi173 ceph-mon[26787]: pgmap v212: 132 pgs: 8 creating+peering, 117 active+clean, 7 unknown; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 KiB/s wr, 63 op/s 2024-05-15T12:37:20.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:20 smithi102 ceph-mon[24799]: pgmap v212: 132 pgs: 8 creating+peering, 117 active+clean, 7 unknown; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 KiB/s wr, 63 op/s 2024-05-15T12:37:20.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:20 smithi102 ceph-mon[31347]: pgmap v212: 132 pgs: 8 creating+peering, 117 active+clean, 7 unknown; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 2.7 KiB/s wr, 63 op/s 2024-05-15T12:37:20.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:20 smithi173 systemd[1]: Starting Ceph iscsi.iscsi.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:37:20.544 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled prometheus update... 2024-05-15T12:37:20.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:20 smithi173 podman[46590]: 2024-05-15 12:37:20.443309909 +0000 UTC m=+0.017019923 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:37:20.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:20 smithi173 podman[46590]: 2024-05-15 12:37:20.563656063 +0000 UTC m=+0.137366151 container create 0365b95a94001275ca54202c28b58979bb4661d9f4c9cc03d4eda55ebe0a70f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:37:21.259 DEBUG:teuthology.orchestra.run.smithi173:prometheus.a> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@prometheus.a.service 2024-05-15T12:37:21.262 INFO:tasks.cephadm:Adding node-exporter.a on smithi102 2024-05-15T12:37:21.262 INFO:tasks.cephadm:Adding node-exporter.b on smithi173 2024-05-15T12:37:21.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply node-exporter '2;smithi102=a;smithi173=b' 2024-05-15T12:37:21.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:21 smithi173 podman[46590]: 2024-05-15 12:37:21.470459103 +0000 UTC m=+1.044169125 container init 0365b95a94001275ca54202c28b58979bb4661d9f4c9cc03d4eda55ebe0a70f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-osd-ec-partial-reads-ae24812, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-05-15T12:37:21.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:21 smithi173 podman[46590]: 2024-05-15 12:37:21.475223249 +0000 UTC m=+1.048933265 container start 0365b95a94001275ca54202c28b58979bb4661d9f4c9cc03d4eda55ebe0a70f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-05-15T12:37:21.666 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:21.892 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:21 smithi173 ceph-mon[26787]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:21.892 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:21 smithi173 ceph-mon[26787]: Saving service prometheus spec with placement smithi173=a;count:1 2024-05-15T12:37:21.892 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:21 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:21.892 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:21 smithi173 ceph-mon[26787]: osdmap e68: 8 total, 8 up, 8 in 2024-05-15T12:37:21.892 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:21 smithi173 ceph-mon[26787]: pgmap v214: 132 pgs: 132 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.3 KiB/s wr, 166 op/s 2024-05-15T12:37:21.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:21 smithi173 bash[46590]: 0365b95a94001275ca54202c28b58979bb4661d9f4c9cc03d4eda55ebe0a70f8 2024-05-15T12:37:21.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:21 smithi173 systemd[1]: Started Ceph iscsi.iscsi.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:37:21.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug Started the configuration object watcher 2024-05-15T12:37:21.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug Checking for config object changes every 1s 2024-05-15T12:37:21.893 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug Processing osd blocklist entries for this node 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[24799]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[24799]: Saving service prometheus spec with placement smithi173=a;count:1 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[24799]: osdmap e68: 8 total, 8 up, 8 in 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[24799]: pgmap v214: 132 pgs: 132 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.3 KiB/s wr, 166 op/s 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[31347]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[31347]: Saving service prometheus spec with placement smithi173=a;count:1 2024-05-15T12:37:21.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:21.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[31347]: osdmap e68: 8 total, 8 up, 8 in 2024-05-15T12:37:21.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:21 smithi102 ceph-mon[31347]: pgmap v214: 132 pgs: 132 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.3 KiB/s wr, 166 op/s 2024-05-15T12:37:22.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug Reading the configuration object to update local LIO configuration 2024-05-15T12:37:22.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug Configuration does not have an entry for this host(smithi173.front.sepia.ceph.com) - nothing to define to LIO 2024-05-15T12:37:22.206 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-05-15T12:37:22.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: Checking pool "datapool" exists for service iscsi.datapool 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: Adding iSCSI gateway http://:@172.21.15.173:5000 to Dashboard 2024-05-15T12:37:22.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi173"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi173"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: Checking dashboard <-> RGW credentials 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/1812542834' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-15T12:37:22.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:22 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: Checking pool "datapool" exists for service iscsi.datapool 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: Adding iSCSI gateway http://:@172.21.15.173:5000 to Dashboard 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-15T12:37:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi173"}]: dispatch 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi173"}]: dispatch 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: Checking dashboard <-> RGW credentials 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/1812542834' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.904 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: Checking pool "datapool" exists for service iscsi.datapool 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: Adding iSCSI gateway http://:@172.21.15.173:5000 to Dashboard 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-05-15T12:37:22.905 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi173"}]: dispatch 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi173"}]: dispatch 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: Checking dashboard <-> RGW credentials 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/1812542834' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-15T12:37:22.906 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:22 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:23.783 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:23 smithi173 ceph-mon[26787]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:23.784 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:23 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-15T12:37:23.784 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:23 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-05-15T12:37:23.784 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:23 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-05-15T12:37:23.784 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:23 smithi173 ceph-mon[26787]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.5 KiB/s wr, 142 op/s 2024-05-15T12:37:23.784 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:23 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:23.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[24799]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:23.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-15T12:37:23.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-05-15T12:37:23.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-05-15T12:37:23.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[24799]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.5 KiB/s wr, 142 op/s 2024-05-15T12:37:23.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:23.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[31347]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:23.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-05-15T12:37:23.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-05-15T12:37:23.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-05-15T12:37:23.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[31347]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 4.5 KiB/s wr, 142 op/s 2024-05-15T12:37:23.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:23 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:24.354 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled node-exporter update... 2024-05-15T12:37:24.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-15T12:37:24.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-05-15T12:37:24.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-05-15T12:37:24.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: osdmap e69: 8 total, 8 up, 8 in 2024-05-15T12:37:24.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: mgrmap e16: y(active, since 5m), standbys: x 2024-05-15T12:37:24.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:24.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:24.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:24.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:24.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:24 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-15T12:37:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-05-15T12:37:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-05-15T12:37:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: osdmap e69: 8 total, 8 up, 8 in 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: mgrmap e16: y(active, since 5m), standbys: x 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: osdmap e69: 8 total, 8 up, 8 in 2024-05-15T12:37:25.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: mgrmap e16: y(active, since 5m), standbys: x 2024-05-15T12:37:25.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:25.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:25.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:25.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:25.154 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:24 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:25.405 DEBUG:teuthology.orchestra.run.smithi102:node-exporter.a> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.a.service 2024-05-15T12:37:25.407 DEBUG:teuthology.orchestra.run.smithi173:node-exporter.b> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.b.service 2024-05-15T12:37:25.410 INFO:tasks.cephadm:Adding alertmanager.a on smithi102 2024-05-15T12:37:25.410 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply alertmanager '1;smithi102=a' 2024-05-15T12:37:25.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[31347]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:25.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[31347]: Saving service node-exporter spec with placement smithi102=a;smithi173=b;count:2 2024-05-15T12:37:25.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[31347]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:25.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[31347]: osdmap e70: 8 total, 8 up, 8 in 2024-05-15T12:37:25.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[31347]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 3.8 KiB/s wr, 128 op/s 2024-05-15T12:37:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[24799]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[24799]: Saving service node-exporter spec with placement smithi102=a;smithi173=b;count:2 2024-05-15T12:37:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[24799]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[24799]: osdmap e70: 8 total, 8 up, 8 in 2024-05-15T12:37:25.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:25 smithi102 ceph-mon[24799]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 3.8 KiB/s wr, 128 op/s 2024-05-15T12:37:25.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:25 smithi173 ceph-mon[26787]: from='client.24457 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi102=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:25.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:25 smithi173 ceph-mon[26787]: Saving service node-exporter spec with placement smithi102=a;smithi173=b;count:2 2024-05-15T12:37:25.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:25 smithi173 ceph-mon[26787]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:25.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:25 smithi173 ceph-mon[26787]: osdmap e70: 8 total, 8 up, 8 in 2024-05-15T12:37:25.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:25 smithi173 ceph-mon[26787]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 3.8 KiB/s wr, 128 op/s 2024-05-15T12:37:26.669 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:28 smithi173 ceph-mon[26787]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 839 B/s wr, 31 op/s; 0 B/s, 0 objects/s recovering 2024-05-15T12:37:28.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:28 smithi102 ceph-mon[31347]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 839 B/s wr, 31 op/s; 0 B/s, 0 objects/s recovering 2024-05-15T12:37:28.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:28 smithi102 ceph-mon[24799]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 839 B/s wr, 31 op/s; 0 B/s, 0 objects/s recovering 2024-05-15T12:37:29.175 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:29 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:29.176 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:29 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:29 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:29.725 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled alertmanager update... 2024-05-15T12:37:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:30 smithi173 ceph-mon[26787]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 511 B/s wr, 19 op/s; 27 B/s, 1 objects/s recovering 2024-05-15T12:37:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:30 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.274 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[31347]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 511 B/s wr, 19 op/s; 27 B/s, 1 objects/s recovering 2024-05-15T12:37:30.274 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.274 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[24799]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 511 B/s wr, 19 op/s; 27 B/s, 1 objects/s recovering 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:30.275 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:30 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:30.847 DEBUG:teuthology.orchestra.run.smithi102:alertmanager.a> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@alertmanager.a.service 2024-05-15T12:37:30.850 INFO:tasks.cephadm:Adding grafana.a on smithi173 2024-05-15T12:37:30.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph orch apply grafana '1;smithi173=a' 2024-05-15T12:37:31.100 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:31.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:31.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-05-15T12:37:31.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:31.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:31.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:31.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:31 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='client.24463 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi102=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:31.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: Saving service alertmanager spec with placement smithi102=a;count:1 2024-05-15T12:37:31.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: Metadata not up to date on all hosts. Skipping non agent specs 2024-05-15T12:37:31.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:31.411 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:31.412 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:31.412 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:31.412 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:31 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[31347]: Deploying daemon prometheus.a on smithi173 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[31347]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 127 B/s wr, 3 op/s; 27 B/s, 1 objects/s recovering 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[24799]: Deploying daemon prometheus.a on smithi173 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[24799]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 127 B/s wr, 3 op/s; 27 B/s, 1 objects/s recovering 2024-05-15T12:37:32.350 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.351 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:32 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.488 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:32 smithi173 ceph-mon[26787]: Deploying daemon prometheus.a on smithi173 2024-05-15T12:37:32.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:32 smithi173 ceph-mon[26787]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 127 B/s wr, 3 op/s; 27 B/s, 1 objects/s recovering 2024-05-15T12:37:32.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:32.489 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:32 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:33.044 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled grafana update... 2024-05-15T12:37:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:33 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:33.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:33 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:33.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:33 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:33.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:33 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:33.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:33 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:33.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:33 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:34.234 DEBUG:teuthology.orchestra.run.smithi173:grafana.a> sudo journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@grafana.a.service 2024-05-15T12:37:34.237 INFO:tasks.cephadm:Setting up client nodes... 2024-05-15T12:37:34.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T12:37:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:34 smithi102 ceph-mon[31347]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 110 B/s wr, 3 op/s; 23 B/s, 1 objects/s recovering 2024-05-15T12:37:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:34 smithi102 ceph-mon[31347]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:34.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:34 smithi102 ceph-mon[31347]: Saving service grafana spec with placement smithi173=a;count:1 2024-05-15T12:37:34.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:34 smithi102 ceph-mon[24799]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 110 B/s wr, 3 op/s; 23 B/s, 1 objects/s recovering 2024-05-15T12:37:34.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:34 smithi102 ceph-mon[24799]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:34.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:34 smithi102 ceph-mon[24799]: Saving service grafana spec with placement smithi173=a;count:1 2024-05-15T12:37:34.489 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:34.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:34 smithi173 ceph-mon[26787]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 110 B/s wr, 3 op/s; 23 B/s, 1 objects/s recovering 2024-05-15T12:37:34.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:34 smithi173 ceph-mon[26787]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:37:34.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:34 smithi173 ceph-mon[26787]: Saving service grafana spec with placement smithi173=a;count:1 2024-05-15T12:37:36.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:36 smithi102 ceph-mon[31347]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 99 B/s wr, 2 op/s; 21 B/s, 1 objects/s recovering 2024-05-15T12:37:36.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:36 smithi102 ceph-mon[24799]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 99 B/s wr, 2 op/s; 21 B/s, 1 objects/s recovering 2024-05-15T12:37:36.487 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-05-15T12:37:36.487 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCQrERmt9SaHRAA/bKRG4pE9CsYSz+4fg0/YQ== 2024-05-15T12:37:36.493 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:36 smithi173 ceph-mon[26787]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 99 B/s wr, 2 op/s; 21 B/s, 1 objects/s recovering 2024-05-15T12:37:37.165 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:37:37.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-15T12:37:37.165 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-15T12:37:37.204 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-15T12:37:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:37 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2955036499' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:37.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:37 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2955036499' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:37:37.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:37 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2955036499' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:37.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:37 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2955036499' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:37:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:37 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2955036499' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:37.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:37 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2955036499' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:37:38.104 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.b/config 2024-05-15T12:37:38.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:38 smithi102 ceph-mon[31347]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-05-15T12:37:38.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:38 smithi102 ceph-mon[24799]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-05-15T12:37:38.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:38 smithi173 ceph-mon[26787]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-05-15T12:37:40.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:40 smithi102 ceph-mon[24799]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-05-15T12:37:40.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:40 smithi102 ceph-mon[31347]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-05-15T12:37:40.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:40 smithi173 ceph-mon[26787]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-05-15T12:37:41.994 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:37:42.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:42 smithi102 ceph-mon[24799]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:37:42.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:42 smithi102 ceph-mon[31347]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:37:42.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:42 smithi173 ceph-mon[26787]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:37:42.560 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-05-15T12:37:42.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCWrERmvSTiIRAA1J5bbaPCompVozNF+l/U+Q== 2024-05-15T12:37:43.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:43 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:43.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:43 smithi173 ceph-mon[26787]: from='client.? 172.21.15.173:0/3449967541' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:43.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:43 smithi173 ceph-mon[26787]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:43.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:43 smithi173 ceph-mon[26787]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:37:43.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:43.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[24799]: from='client.? 172.21.15.173:0/3449967541' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:43.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[24799]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:43.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[24799]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:37:43.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:43.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[31347]: from='client.? 172.21.15.173:0/3449967541' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:43.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[31347]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-15T12:37:43.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:43 smithi102 ceph-mon[31347]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-15T12:37:43.494 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:43 smithi173 systemd[1]: Starting Ceph prometheus.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:37:43.536 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:37:43.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-05-15T12:37:43.536 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-15T12:37:43.577 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-15T12:37:43.577 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-15T12:37:43.577 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-15T12:37:43.760 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:43 smithi173 podman[48678]: 2024-05-15 12:37:43.650856708 +0000 UTC m=+0.018215552 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-05-15T12:37:43.761 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:43 smithi173 podman[48678]: 2024-05-15 12:37:43.772162803 +0000 UTC m=+0.139521656 container create 8b964ad822b990a674d698c7f544391eb423aa55791cc1e38006220e9821e7bf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:37:43.828 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:44.027 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:43 smithi173 podman[48678]: 2024-05-15 12:37:43.982634221 +0000 UTC m=+0.349993054 container init 8b964ad822b990a674d698c7f544391eb423aa55791cc1e38006220e9821e7bf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:37:44.027 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:43 smithi173 podman[48678]: 2024-05-15 12:37:43.985570841 +0000 UTC m=+0.352929684 container start 8b964ad822b990a674d698c7f544391eb423aa55791cc1e38006220e9821e7bf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.011Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.011Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.011Z caller=main.go:570 level=info host_details="(Linux 5.14.0-446.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 3 16:10:18 UTC 2024 x86_64 smithi173 (none))" 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.011Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.011Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.012Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.013Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.013Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.013Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.016Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.016Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.74µs 2024-05-15T12:37:44.028 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.016Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.017Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.017Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.215µs wal_replay_duration=298.785µs wbl_replay_duration=89ns total_replay_duration=333.762µs 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.017Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.017Z caller=main.go:1029 level=info msg="TSDB started" 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.017Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.038Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.569053ms db_storage=1.07µs remote_storage=1.125µs web_handler=287ns query_engine=554ns scrape=436.758µs scrape_sd=91.915µs notify=935ns notify_sd=1.795µs rules=19.794786ms tracing=5.984µs 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.038Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-05-15T12:37:44.029 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:37:44.038Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-05-15T12:37:44.251 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:44 smithi102 ceph-mon[31347]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:37:44.251 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:44 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:44.251 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:44 smithi102 ceph-mon[24799]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:37:44.252 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:44 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:44.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:44 smithi173 ceph-mon[26787]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:37:44.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:44 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:44.294 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 bash[48678]: 8b964ad822b990a674d698c7f544391eb423aa55791cc1e38006220e9821e7bf 2024-05-15T12:37:44.294 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:37:44 smithi173 systemd[1]: Started Ceph prometheus.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:37:45.210 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:45.211 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:45.211 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:45 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-15T12:37:45.211 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ignoring --setuser ceph since I am not root 2024-05-15T12:37:45.211 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ignoring --setgroup ceph since I am not root 2024-05-15T12:37:45.211 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:45.212 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:45.212 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:45 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-15T12:37:45.220 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:45.220 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' 2024-05-15T12:37:45.220 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:45 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-05-15T12:37:45.221 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:45 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: ignoring --setuser ceph since I am not root 2024-05-15T12:37:45.221 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:45 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: ignoring --setgroup ceph since I am not root 2024-05-15T12:37:45.221 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:45 smithi173 ceph-mgr[28984]: -- 172.21.15.173:0/2358293511 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dccd2d25a0 con 0x55dccd2a6800 2024-05-15T12:37:45.486 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:45 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:45.309+0000 7f3a54e26200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:37:45.493 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:45.313+0000 7fa457543200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-15T12:37:45.493 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:45.503+0000 7fa457543200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:37:45.734 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:37:45.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:45 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:45.496+0000 7f3a54e26200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:37:45.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:45 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:45.604+0000 7f3a54e26200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:37:45.746 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:45.609+0000 7fa457543200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-15T12:37:46.008 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:45.931+0000 7fa457543200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:37:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:46 smithi173 ceph-mon[26787]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-15T12:37:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:46 smithi173 ceph-mon[26787]: mgrmap e17: y(active, since 5m), standbys: x 2024-05-15T12:37:46.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:46 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/1159353508' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T12:37:46.245 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:45 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:45.928+0000 7f3a54e26200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-15T12:37:46.245 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:46 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:46.017+0000 7f3a54e26200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:37:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:46 smithi102 ceph-mon[24799]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-15T12:37:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:46 smithi102 ceph-mon[24799]: mgrmap e17: y(active, since 5m), standbys: x 2024-05-15T12:37:46.402 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:46 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1159353508' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T12:37:46.403 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:46 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:46.018+0000 7fa457543200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-15T12:37:46.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:46 smithi102 ceph-mon[31347]: from='mgr.14152 172.21.15.102:0/2050005085' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-05-15T12:37:46.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:46 smithi102 ceph-mon[31347]: mgrmap e17: y(active, since 5m), standbys: x 2024-05-15T12:37:46.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:46 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/1159353508' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-15T12:37:46.481 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":2035708645},{"type":"v1","addr":"172.21.15.102:6801","nonce":2035708645}]},"active_addr":"172.21.15.102:6801/2035708645","active_change":"2024-05-15T12:32:22.919872+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.102:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3916506932}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":27429583}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":688628141}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3820605158}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":308960120}]}]} 2024-05-15T12:37:46.488 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-15T12:37:46.488 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-15T12:37:46.488 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T12:37:46.741 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:47.329 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:47 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:47.237+0000 7f3a54e26200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:37:47.425 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:47 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:47.264+0000 7fa457543200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-15T12:37:47.635 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:47 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:47.449+0000 7f3a54e26200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:37:47.895 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:47 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:47.435+0000 7fa457543200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-15T12:37:47.896 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:47 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:47.610+0000 7fa457543200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:37:47.896 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:47 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:47.906+0000 7fa457543200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:37:47.955 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:47 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:47.645+0000 7f3a54e26200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-15T12:37:48.152 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:48 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:48.008+0000 7fa457543200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:37:48.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:47 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:47.965+0000 7f3a54e26200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-15T12:37:48.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:48 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:48.060+0000 7f3a54e26200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-15T12:37:48.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:37:48.622 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":70,"fsid":"eac36fa0-12b6-11ef-bc98-c7b262605968","created":"2024-05-15T12:31:20.823409+0000","modified":"2024-05-15T12:37:24.653280+0000","last_up_change":"2024-05-15T12:36:55.840485+0000","last_in_change":"2024-05-15T12:36:36.892145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:34:45.237868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-05-15T12:37:07.277102+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-05-15T12:37:07.522543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-05-15T12:37:09.371964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-05-15T12:37:11.328507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-05-15T12:37:13.381512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a157badc-d604-423e-80fd-7205437c36de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6803","nonce":4160180593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6805","nonce":4160180593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6809","nonce":4160180593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6807","nonce":4160180593}]},"public_addr":"172.21.15.102:6803/4160180593","cluster_addr":"172.21.15.102:6805/4160180593","heartbeat_back_addr":"172.21.15.102:6809/4160180593","heartbeat_front_addr":"172.21.15.102:6807/4160180593","state":["exists","up"]},{"osd":1,"uuid":"3cfe8e1f-978d-4e11-94f3-094a13ac15fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6811","nonce":838755088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6813","nonce":838755088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6817","nonce":838755088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6815","nonce":838755088}]},"public_addr":"172.21.15.102:6811/838755088","cluster_addr":"172.21.15.102:6813/838755088","heartbeat_back_addr":"172.21.15.102:6817/838755088","heartbeat_front_addr":"172.21.15.102:6815/838755088","state":["exists","up"]},{"osd":2,"uuid":"f49f2eb3-557b-456c-91ab-7098fd58995a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6819","nonce":1924240300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6821","nonce":1924240300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6825","nonce":1924240300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6823","nonce":1924240300}]},"public_addr":"172.21.15.102:6819/1924240300","cluster_addr":"172.21.15.102:6821/1924240300","heartbeat_back_addr":"172.21.15.102:6825/1924240300","heartbeat_front_addr":"172.21.15.102:6823/1924240300","state":["exists","up"]},{"osd":3,"uuid":"f31ac056-226b-4796-80c3-4111dad7b9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6827","nonce":1700281993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6829","nonce":1700281993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6833","nonce":1700281993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6831","nonce":1700281993}]},"public_addr":"172.21.15.102:6827/1700281993","cluster_addr":"172.21.15.102:6829/1700281993","heartbeat_back_addr":"172.21.15.102:6833/1700281993","heartbeat_front_addr":"172.21.15.102:6831/1700281993","state":["exists","up"]},{"osd":4,"uuid":"1bf0a066-545a-4f0d-8fef-a4d9136d4e69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6801","nonce":1901441343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6803","nonce":1901441343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6807","nonce":1901441343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6805","nonce":1901441343}]},"public_addr":"172.21.15.173:6801/1901441343","cluster_addr":"172.21.15.173:6803/1901441343","heartbeat_back_addr":"172.21.15.173:6807/1901441343","heartbeat_front_addr":"172.21.15.173:6805/1901441343","state":["exists","up"]},{"osd":5,"uuid":"1b7b81d1-9878-40a9-a805-ce2b8fcd3e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6809","nonce":2038365521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6811","nonce":2038365521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6815","nonce":2038365521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6813","nonce":2038365521}]},"public_addr":"172.21.15.173:6809/2038365521","cluster_addr":"172.21.15.173:6811/2038365521","heartbeat_back_addr":"172.21.15.173:6815/2038365521","heartbeat_front_addr":"172.21.15.173:6813/2038365521","state":["exists","up"]},{"osd":6,"uuid":"5c5cada8-7d3c-40d7-b215-33d221ad63ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6817","nonce":2130176336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6819","nonce":2130176336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6823","nonce":2130176336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6821","nonce":2130176336}]},"public_addr":"172.21.15.173:6817/2130176336","cluster_addr":"172.21.15.173:6819/2130176336","heartbeat_back_addr":"172.21.15.173:6823/2130176336","heartbeat_front_addr":"172.21.15.173:6821/2130176336","state":["exists","up"]},{"osd":7,"uuid":"fd4b9b4c-13f2-49ac-8106-bcbc89a7b765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6825","nonce":467039878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6827","nonce":467039878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6831","nonce":467039878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6829","nonce":467039878}]},"public_addr":"172.21.15.173:6825/467039878","cluster_addr":"172.21.15.173:6827/467039878","heartbeat_back_addr":"172.21.15.173:6831/467039878","heartbeat_front_addr":"172.21.15.173:6829/467039878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:33:51.093575+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:34:16.553531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:34:42.922376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:35:11.581849+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:35:35.862174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:02.347318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:29.113689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:54.372914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/2982719131":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/785097316":"2024-05-16T12:32:22.919753+0000","172.21.15.102:6801/3724371257":"2024-05-16T12:32:22.919753+0000","172.21.15.102:6800/3724371257":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/4275670978":"2024-05-16T12:31:56.860756+0000","172.21.15.102:0/2007662139":"2024-05-16T12:31:56.860756+0000","172.21.15.102:6800/455878624":"2024-05-16T12:31:56.860756+0000","172.21.15.102:6801/455878624":"2024-05-16T12:31:56.860756+0000","172.21.15.102:0/2850653973":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/2344980692":"2024-05-16T12:31:56.860756+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:37:48.623 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:48 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:48.219+0000 7fa457543200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:37:48.623 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:48 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:48.425+0000 7fa457543200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:37:48.673 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:48 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:48.276+0000 7f3a54e26200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-15T12:37:48.673 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:48 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:48.474+0000 7f3a54e26200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-15T12:37:48.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:48 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2655015700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:37:48.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:48 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2655015700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:37:48.992 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:48 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2655015700' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:37:49.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:49.150+0000 7f3a54e26200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:37:49.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:49 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:49.056+0000 7fa457543200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-15T12:37:49.742 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:49 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:49.492+0000 7fa457543200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:37:49.743 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:49 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:49.591+0000 7fa457543200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:37:49.890 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:49.600+0000 7f3a54e26200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-15T12:37:49.891 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:49.699+0000 7f3a54e26200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-15T12:37:50.114 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:49 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:49.798+0000 7fa457543200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:37:50.116 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-15T12:37:50.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-15T12:37:50.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:49.909+0000 7f3a54e26200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-15T12:37:50.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:50.342+0000 7fa457543200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:37:50.559 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:50.563 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:50 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:50.480+0000 7f3a54e26200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-15T12:37:50.700 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:50.438+0000 7fa457543200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:37:50.941 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:50 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:50.573+0000 7f3a54e26200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-15T12:37:50.941 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:50 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:50.818+0000 7f3a54e26200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:37:50.941 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:50 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:50.951+0000 7f3a54e26200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:37:50.979 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:50.710+0000 7fa457543200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-15T12:37:50.979 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:50.850+0000 7fa457543200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-15T12:37:50.979 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:50 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:50.950+0000 7fa457543200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:37:51.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:51.048+0000 7f3a54e26200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-15T12:37:51.244 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:51.167+0000 7f3a54e26200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:37:51.303 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:51 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:51.079+0000 7fa457543200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-15T12:37:51.652 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:51 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:51.395+0000 7fa457543200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:37:51.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:51.475+0000 7f3a54e26200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-15T12:37:52.199 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:52.077+0000 7fa457543200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:37:52.345 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:52.076+0000 7f3a54e26200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-15T12:37:52.345 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:52.298+0000 7f3a54e26200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:37:52.345 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: [15/May/2024:12:37:52] ENGINE Bus STARTING 2024-05-15T12:37:52.449 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:52.311+0000 7fa457543200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-15T12:37:52.449 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:37:52] ENGINE Bus STARTING 2024-05-15T12:37:52.449 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: Standby manager daemon x restarted 2024-05-15T12:37:52.449 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: Standby manager daemon x started 2024-05-15T12:37:52.449 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:37:52.449 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: Active manager daemon y restarted 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: Activating manager daemon y 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: osdmap e71: 8 total, 8 up, 8 in 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: mgrmap e18: y(active, starting, since 0.00929906s), standbys: x 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:37:52.450 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[24799]: Manager daemon y is now available 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: Standby manager daemon x restarted 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: Standby manager daemon x started 2024-05-15T12:37:52.451 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: Active manager daemon y restarted 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: Activating manager daemon y 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: osdmap e71: 8 total, 8 up, 8 in 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: mgrmap e18: y(active, starting, since 0.00929906s), standbys: x 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:37:52.452 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:37:52.453 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:52 smithi102 ceph-mon[31347]: Manager daemon y is now available 2024-05-15T12:37:52.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: CherryPy Checker: 2024-05-15T12:37:52.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: The Application mounted at '' has an empty config. 2024-05-15T12:37:52.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: 2024-05-15T12:37:52.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: [15/May/2024:12:37:52] ENGINE Serving on http://:::9283 2024-05-15T12:37:52.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:37:52 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x[28980]: [15/May/2024:12:37:52] ENGINE Bus STARTED 2024-05-15T12:37:52.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: Standby manager daemon x restarted 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: Standby manager daemon x started 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.? 172.21.15.173:0/1046862618' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: Active manager daemon y restarted 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: Activating manager daemon y 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: osdmap e71: 8 total, 8 up, 8 in 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: mgrmap e18: y(active, starting, since 0.00929906s), standbys: x 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-15T12:37:52.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-15T12:37:52.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:52 smithi173 ceph-mon[26787]: Manager daemon y is now available 2024-05-15T12:37:52.788 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: CherryPy Checker: 2024-05-15T12:37:52.789 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: The Application mounted at '' has an empty config. 2024-05-15T12:37:52.789 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:37:52.789 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:37:52] ENGINE Serving on http://:::9283 2024-05-15T12:37:52.789 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:52 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:37:52] ENGINE Bus STARTED 2024-05-15T12:37:53.625 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:37:53 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:37:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]': finished 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]': finished 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:53.626 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: Updating smithi102:/etc/ceph/ceph.conf 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: [15/May/2024:12:37:53] ENGINE Bus STARTING 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[31347]: mgrmap e19: y(active, since 1.0137s), standbys: x 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:53.627 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]': finished 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]': finished 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: Updating smithi102:/etc/ceph/ceph.conf 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:37:53.628 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:37:53.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:37:53.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:37:53.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:37:53.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:37:53.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: [15/May/2024:12:37:53] ENGINE Bus STARTING 2024-05-15T12:37:53.629 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:53 smithi102 ceph-mon[24799]: mgrmap e19: y(active, since 1.0137s), standbys: x 2024-05-15T12:37:53.655 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:37:53.655 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":71,"fsid":"eac36fa0-12b6-11ef-bc98-c7b262605968","created":"2024-05-15T12:31:20.823409+0000","modified":"2024-05-15T12:37:52.313431+0000","last_up_change":"2024-05-15T12:36:55.840485+0000","last_in_change":"2024-05-15T12:36:36.892145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-15T12:34:45.237868+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-05-15T12:37:07.277102+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-05-15T12:37:07.522543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-05-15T12:37:09.371964+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-05-15T12:37:11.328507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-05-15T12:37:13.381512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a157badc-d604-423e-80fd-7205437c36de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6803","nonce":4160180593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6805","nonce":4160180593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6809","nonce":4160180593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":4160180593},{"type":"v1","addr":"172.21.15.102:6807","nonce":4160180593}]},"public_addr":"172.21.15.102:6803/4160180593","cluster_addr":"172.21.15.102:6805/4160180593","heartbeat_back_addr":"172.21.15.102:6809/4160180593","heartbeat_front_addr":"172.21.15.102:6807/4160180593","state":["exists","up"]},{"osd":1,"uuid":"3cfe8e1f-978d-4e11-94f3-094a13ac15fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6811","nonce":838755088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6813","nonce":838755088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6817","nonce":838755088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":838755088},{"type":"v1","addr":"172.21.15.102:6815","nonce":838755088}]},"public_addr":"172.21.15.102:6811/838755088","cluster_addr":"172.21.15.102:6813/838755088","heartbeat_back_addr":"172.21.15.102:6817/838755088","heartbeat_front_addr":"172.21.15.102:6815/838755088","state":["exists","up"]},{"osd":2,"uuid":"f49f2eb3-557b-456c-91ab-7098fd58995a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6819","nonce":1924240300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6821","nonce":1924240300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6825","nonce":1924240300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1924240300},{"type":"v1","addr":"172.21.15.102:6823","nonce":1924240300}]},"public_addr":"172.21.15.102:6819/1924240300","cluster_addr":"172.21.15.102:6821/1924240300","heartbeat_back_addr":"172.21.15.102:6825/1924240300","heartbeat_front_addr":"172.21.15.102:6823/1924240300","state":["exists","up"]},{"osd":3,"uuid":"f31ac056-226b-4796-80c3-4111dad7b9fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6827","nonce":1700281993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6829","nonce":1700281993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6833","nonce":1700281993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":1700281993},{"type":"v1","addr":"172.21.15.102:6831","nonce":1700281993}]},"public_addr":"172.21.15.102:6827/1700281993","cluster_addr":"172.21.15.102:6829/1700281993","heartbeat_back_addr":"172.21.15.102:6833/1700281993","heartbeat_front_addr":"172.21.15.102:6831/1700281993","state":["exists","up"]},{"osd":4,"uuid":"1bf0a066-545a-4f0d-8fef-a4d9136d4e69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6801","nonce":1901441343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6803","nonce":1901441343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6807","nonce":1901441343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1901441343},{"type":"v1","addr":"172.21.15.173:6805","nonce":1901441343}]},"public_addr":"172.21.15.173:6801/1901441343","cluster_addr":"172.21.15.173:6803/1901441343","heartbeat_back_addr":"172.21.15.173:6807/1901441343","heartbeat_front_addr":"172.21.15.173:6805/1901441343","state":["exists","up"]},{"osd":5,"uuid":"1b7b81d1-9878-40a9-a805-ce2b8fcd3e83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6809","nonce":2038365521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6811","nonce":2038365521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6815","nonce":2038365521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2038365521},{"type":"v1","addr":"172.21.15.173:6813","nonce":2038365521}]},"public_addr":"172.21.15.173:6809/2038365521","cluster_addr":"172.21.15.173:6811/2038365521","heartbeat_back_addr":"172.21.15.173:6815/2038365521","heartbeat_front_addr":"172.21.15.173:6813/2038365521","state":["exists","up"]},{"osd":6,"uuid":"5c5cada8-7d3c-40d7-b215-33d221ad63ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6817","nonce":2130176336}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6819","nonce":2130176336}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6823","nonce":2130176336}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2130176336},{"type":"v1","addr":"172.21.15.173:6821","nonce":2130176336}]},"public_addr":"172.21.15.173:6817/2130176336","cluster_addr":"172.21.15.173:6819/2130176336","heartbeat_back_addr":"172.21.15.173:6823/2130176336","heartbeat_front_addr":"172.21.15.173:6821/2130176336","state":["exists","up"]},{"osd":7,"uuid":"fd4b9b4c-13f2-49ac-8106-bcbc89a7b765","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6825","nonce":467039878}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6827","nonce":467039878}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6831","nonce":467039878}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":467039878},{"type":"v1","addr":"172.21.15.173:6829","nonce":467039878}]},"public_addr":"172.21.15.173:6825/467039878","cluster_addr":"172.21.15.173:6827/467039878","heartbeat_back_addr":"172.21.15.173:6831/467039878","heartbeat_front_addr":"172.21.15.173:6829/467039878","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:33:51.093575+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:34:16.553531+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:34:42.922376+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:35:11.581849+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:35:35.862174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:02.347318+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:29.113689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-15T12:36:54.372914+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:0/3820605158":"2024-05-16T12:37:52.313399+0000","172.21.15.102:0/2344980692":"2024-05-16T12:31:56.860756+0000","172.21.15.102:0/688628141":"2024-05-16T12:37:52.313399+0000","172.21.15.102:0/2850653973":"2024-05-16T12:32:22.919753+0000","172.21.15.102:6800/455878624":"2024-05-16T12:31:56.860756+0000","172.21.15.102:6801/455878624":"2024-05-16T12:31:56.860756+0000","172.21.15.102:6801/2035708645":"2024-05-16T12:37:52.313399+0000","172.21.15.102:0/4275670978":"2024-05-16T12:31:56.860756+0000","172.21.15.102:6800/3724371257":"2024-05-16T12:32:22.919753+0000","172.21.15.102:6801/3724371257":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/3916506932":"2024-05-16T12:37:52.313399+0000","172.21.15.102:6800/2035708645":"2024-05-16T12:37:52.313399+0000","172.21.15.102:0/2007662139":"2024-05-16T12:31:56.860756+0000","172.21.15.102:0/785097316":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/2982719131":"2024-05-16T12:32:22.919753+0000","172.21.15.102:0/308960120":"2024-05-16T12:37:52.313399+0000","172.21.15.102:0/27429583":"2024-05-16T12:37:52.313399+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-15T12:37:53.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:37:53 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:37:53.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:53.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:53.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:37:53.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]': finished 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]': finished 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: Updating smithi102:/etc/ceph/ceph.conf 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: Updating smithi173:/etc/ceph/ceph.conf 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-05-15T12:37:53.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:37:53.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-15T12:37:53.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: [15/May/2024:12:37:53] ENGINE Bus STARTING 2024-05-15T12:37:53.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:53 smithi173 ceph-mon[26787]: mgrmap e19: y(active, since 1.0137s), standbys: x 2024-05-15T12:37:54.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-15T12:37:54.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-15T12:37:54.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-15T12:37:54.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-15T12:37:54.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-15T12:37:54.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-15T12:37:54.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-15T12:37:54.424 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-15T12:37:54.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: [15/May/2024:12:37:53] ENGINE Serving on https://172.21.15.102:7151 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: [15/May/2024:12:37:53] ENGINE Serving on http://172.21.15.102:8765 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: [15/May/2024:12:37:53] ENGINE Bus STARTED 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/1868442998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:37:54.653 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[31347]: Deploying daemon alertmanager.a on smithi102 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: [15/May/2024:12:37:53] ENGINE Serving on https://172.21.15.102:7151 2024-05-15T12:37:54.654 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: [15/May/2024:12:37:53] ENGINE Serving on http://172.21.15.102:8765 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: [15/May/2024:12:37:53] ENGINE Bus STARTED 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1868442998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.655 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.656 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.656 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:54 smithi102 ceph-mon[24799]: Deploying daemon alertmanager.a on smithi102 2024-05-15T12:37:54.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:37:54.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-05-15T12:37:54.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: [15/May/2024:12:37:53] ENGINE Serving on https://172.21.15.102:7151 2024-05-15T12:37:54.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:37:54.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:54.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: [15/May/2024:12:37:53] ENGINE Serving on http://172.21.15.102:8765 2024-05-15T12:37:54.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: [15/May/2024:12:37:53] ENGINE Bus STARTED 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: Updating smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: Updating smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/1868442998' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:37:54.745 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:54 smithi173 ceph-mon[26787]: Deploying daemon alertmanager.a on smithi102 2024-05-15T12:37:54.865 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:54.922 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:54.922 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:54.925 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:54.948 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:55.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:55 smithi102 ceph-mon[31347]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:55.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:55 smithi102 ceph-mon[31347]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-15T12:37:55.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:55 smithi102 ceph-mon[24799]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:55.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:55 smithi102 ceph-mon[24799]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-15T12:37:56.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:55 smithi173 ceph-mon[26787]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:56.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:55 smithi173 ceph-mon[26787]: mgrmap e20: y(active, since 2s), standbys: x 2024-05-15T12:37:58.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:57 smithi102 ceph-mon[31347]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:58.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:57 smithi102 ceph-mon[31347]: mgrmap e21: y(active, since 4s), standbys: x 2024-05-15T12:37:58.010 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:57 smithi102 ceph-mon[24799]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:58.010 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:57 smithi102 ceph-mon[24799]: mgrmap e21: y(active, since 4s), standbys: x 2024-05-15T12:37:58.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:57 smithi173 ceph-mon[26787]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:37:58.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:57 smithi173 ceph-mon[26787]: mgrmap e21: y(active, since 4s), standbys: x 2024-05-15T12:37:58.752 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:37:58.753 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:00.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:37:59 smithi102 ceph-mon[31347]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:38:00.143 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:37:59 smithi102 ceph-mon[24799]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:38:00.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:37:59 smithi173 ceph-mon[26787]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-05-15T12:38:02.046 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:01 smithi173 ceph-mon[26787]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-15T12:38:02.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:01 smithi102 ceph-mon[31347]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-15T12:38:02.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:01 smithi102 ceph-mon[24799]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-05-15T12:38:03.216 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:03.475 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:38:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:38:03.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:03 smithi102 ceph-mon[24799]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-15T12:38:03.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:03 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:38:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:38:03.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:03 smithi102 ceph-mon[31347]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-15T12:38:04.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:03 smithi173 ceph-mon[26787]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-05-15T12:38:04.699 INFO:teuthology.orchestra.run.smithi102.stdout:77309411369 2024-05-15T12:38:04.700 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-15T12:38:04.932 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:04 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:05.046 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:04 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:05.046 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:04 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:06.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:05 smithi102 ceph-mon[31347]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-15T12:38:06.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:05 smithi102 ceph-mon[24799]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-15T12:38:06.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:05 smithi173 ceph-mon[26787]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-05-15T12:38:07.769 INFO:teuthology.orchestra.run.smithi102.stdout:55834574896 2024-05-15T12:38:07.770 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-15T12:38:08.047 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:07 smithi102 ceph-mon[24799]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:08.048 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:07 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:08.048 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:07 smithi102 ceph-mon[31347]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:08.048 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:07 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:08.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:07 smithi173 ceph-mon[26787]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:08.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:07 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:08.326 INFO:teuthology.orchestra.run.smithi102.stdout:197568495635 2024-05-15T12:38:08.326 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-15T12:38:09.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:09 smithi102 ceph-mon[31347]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:09.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:09 smithi102 ceph-mon[24799]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:10.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:09 smithi173 ceph-mon[26787]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:10.763 INFO:teuthology.orchestra.run.smithi102.stdout:163208757275 2024-05-15T12:38:10.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-15T12:38:12.101 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:11 smithi102 ceph-mon[24799]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:12.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:11 smithi102 ceph-mon[31347]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:12.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:11 smithi173 ceph-mon[26787]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-05-15T12:38:12.623 INFO:teuthology.orchestra.run.smithi102.stdout:128849018912 2024-05-15T12:38:12.623 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-15T12:38:13.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:38:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:38:13.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:13 smithi102 ceph-mon[31347]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:13.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:13 smithi102 ceph-mon[24799]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:13.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:13 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:38:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:38:13.934 INFO:teuthology.orchestra.run.smithi102.stdout:231928233999 2024-05-15T12:38:13.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-15T12:38:14.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:13 smithi173 ceph-mon[26787]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:14.667 INFO:teuthology.orchestra.run.smithi102.stdout:107374182438 2024-05-15T12:38:14.668 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-15T12:38:15.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:14 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:15.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:14 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:15.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:14 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:16.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:15 smithi102 ceph-mon[31347]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:16.131 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:15 smithi102 ceph-mon[24799]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:15 smithi173 ceph-mon[26787]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:16.634 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:17.081 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:17.105 INFO:teuthology.orchestra.run.smithi102.stdout:34359738422 2024-05-15T12:38:17.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-15T12:38:17.112 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:17.119 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:17.126 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:17.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:17 smithi102 ceph-mon[24799]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:17.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:17 smithi102 ceph-mon[31347]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:18.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:17 smithi173 ceph-mon[26787]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:18.590 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:18.590 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:20.027 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:19 smithi102 ceph-mon[24799]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:20.027 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:19 smithi102 ceph-mon[31347]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:20.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:19 smithi173 ceph-mon[26787]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:21.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:21 smithi102 ceph-mon[31347]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:21.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:21 smithi102 ceph-mon[24799]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:22.112 INFO:teuthology.orchestra.run.smithi102.stdout:55834574899 2024-05-15T12:38:22.161 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:21 smithi173 ceph-mon[26787]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:22.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:22 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2267418420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T12:38:22.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:22 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:22.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:22 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/3484407950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T12:38:23.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:22 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2267418420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T12:38:23.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:22 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:23.129 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:22 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/3484407950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T12:38:23.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:22 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2267418420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-15T12:38:23.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:22 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:23.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:22 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/3484407950' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-15T12:38:23.402 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:23 smithi102 podman[59117]: 2024-05-15 12:38:23.243388161 +0000 UTC m=+1.348745789 container create ed04e8e4535f148a21fb99e2bf46896ecf5ca787fa2b4ff855e6daa8da24e9b4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:38:23.404 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:38:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:38:23.542 INFO:teuthology.orchestra.run.smithi102.stdout:77309411373 2024-05-15T12:38:23.823 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:23 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:38:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:38:24.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:23 smithi102 ceph-mon[31347]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:24.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:23 smithi102 ceph-mon[24799]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:24.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:23 smithi173 ceph-mon[26787]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:24.501 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411373 for osd.2 2024-05-15T12:38:24.501 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:24.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:24 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:25.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:24 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:25.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:24 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:25.652 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 podman[59117]: 2024-05-15 12:38:25.49225397 +0000 UTC m=+3.597611623 container init ed04e8e4535f148a21fb99e2bf46896ecf5ca787fa2b4ff855e6daa8da24e9b4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:38:25.652 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 podman[59117]: 2024-05-15 12:38:25.495935232 +0000 UTC m=+3.601292868 container start ed04e8e4535f148a21fb99e2bf46896ecf5ca787fa2b4ff855e6daa8da24e9b4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.507Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.507Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.509Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.510Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.533Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.533Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.535Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:25.535Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 bash[59117]: ed04e8e4535f148a21fb99e2bf46896ecf5ca787fa2b4ff855e6daa8da24e9b4 2024-05-15T12:38:25.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:25 smithi102 systemd[1]: Started Ceph alertmanager.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:38:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.051 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-15T12:38:26.052 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.052 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:26.052 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:25 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:38:25] ENGINE Bus STOPPING 2024-05-15T12:38:26.053 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:38:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-15T12:38:26.054 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:38:25] ENGINE Bus STOPPED 2024-05-15T12:38:26.054 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:38:25] ENGINE Bus STARTING 2024-05-15T12:38:26.054 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:38:25] ENGINE Serving on http://:::9283 2024-05-15T12:38:26.054 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:25 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:38:25] ENGINE Bus STARTED 2024-05-15T12:38:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:26.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-15T12:38:26.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:25 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:27.035 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:26 smithi102 ceph-mon[24799]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-15T12:38:27.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:26 smithi102 ceph-mon[24799]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-15T12:38:27.036 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:26 smithi102 ceph-mon[24799]: Deploying daemon grafana.a on smithi173 2024-05-15T12:38:27.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:26 smithi102 ceph-mon[31347]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-15T12:38:27.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:26 smithi102 ceph-mon[31347]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-15T12:38:27.036 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:26 smithi102 ceph-mon[31347]: Deploying daemon grafana.a on smithi173 2024-05-15T12:38:27.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:26 smithi173 ceph-mon[26787]: Regenerating cephadm self-signed grafana TLS certificates 2024-05-15T12:38:27.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:26 smithi173 ceph-mon[26787]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-05-15T12:38:27.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:26 smithi173 ceph-mon[26787]: Deploying daemon grafana.a on smithi173 2024-05-15T12:38:27.784 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:27.511Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00098042s 2024-05-15T12:38:28.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:27 smithi102 ceph-mon[24799]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:28.039 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:27 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:28.039 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:27 smithi102 ceph-mon[31347]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:28.039 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:27 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:27 smithi173 ceph-mon[26787]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:28.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:27 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:28.446 INFO:teuthology.orchestra.run.smithi102.stdout:163208757278 2024-05-15T12:38:28.715 INFO:teuthology.orchestra.run.smithi102.stdout:197568495639 2024-05-15T12:38:28.897 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:28 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/4199652845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T12:38:28.897 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:28 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2573401286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T12:38:28.897 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:28 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/4145066504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T12:38:28.897 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:28 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/4199652845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T12:38:28.897 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:28 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2573401286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T12:38:28.897 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:28 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/4145066504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T12:38:29.103 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574896 got 55834574899 for osd.1 2024-05-15T12:38:29.103 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:29.219 INFO:teuthology.orchestra.run.smithi102.stdout:128849018915 2024-05-15T12:38:29.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:28 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/4199652845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-15T12:38:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:28 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2573401286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-15T12:38:29.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:28 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/4145066504' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-15T12:38:30.095 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018912 got 128849018915 for osd.4 2024-05-15T12:38:30.096 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:30.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:29 smithi102 ceph-mon[31347]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:30.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:29 smithi102 ceph-mon[24799]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:30.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:29 smithi173 ceph-mon[26787]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:30.959 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495635 got 197568495639 for osd.6 2024-05-15T12:38:30.987 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:31.547 INFO:teuthology.orchestra.run.smithi102.stdout:107374182442 2024-05-15T12:38:31.773 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:32.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:31 smithi102 ceph-mon[31347]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:32.104 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:31 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/870308510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T12:38:32.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:31 smithi102 ceph-mon[24799]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:32.104 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:31 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/870308510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T12:38:32.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:31 smithi173 ceph-mon[26787]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:32.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:31 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/870308510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-15T12:38:33.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:38:33 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:38:33.849 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:33 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:38:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:38:33.860 INFO:teuthology.orchestra.run.smithi102.stdout:231928234003 2024-05-15T12:38:34.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:33 smithi102 ceph-mon[31347]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:34.135 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:33 smithi102 ceph-mon[24799]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:34.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:33 smithi173 ceph-mon[26787]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:34.886 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757275 got 163208757278 for osd.5 2024-05-15T12:38:34.886 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:35.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:34 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:35.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:34 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/1729739847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T12:38:35.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:34 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:35.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:34 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/1729739847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T12:38:35.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:34 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:35.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:34 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/1729739847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-15T12:38:35.362 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182438 got 107374182442 for osd.3 2024-05-15T12:38:35.362 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:35.652 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:38:35 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:38:35.513Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002909064s 2024-05-15T12:38:36.007 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:35 smithi102 ceph-mon[31347]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:36.007 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:35 smithi102 ceph-mon[24799]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:36.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:35 smithi173 ceph-mon[26787]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:36.452 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233999 got 231928234003 for osd.7 2024-05-15T12:38:36.453 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:38.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:37 smithi102 ceph-mon[24799]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:38.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:37 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:38.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:37 smithi102 ceph-mon[31347]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:38.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:37 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:38.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:37 smithi173 ceph-mon[26787]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:38.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:37 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:38.339 INFO:teuthology.orchestra.run.smithi102.stdout:34359738427 2024-05-15T12:38:38.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:38 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/311579086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T12:38:38.902 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:38 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/311579086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T12:38:39.018 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738422 got 34359738427 for osd.0 2024-05-15T12:38:39.018 DEBUG:teuthology.parallel:result is None 2024-05-15T12:38:39.018 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-15T12:38:39.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:38:39.169 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:38 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/311579086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-15T12:38:39.295 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:39.744 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:39 smithi173 systemd[1]: Starting Ceph grafana.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:38:40.039 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:39 smithi102 ceph-mon[31347]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:40.040 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:39 smithi102 ceph-mon[24799]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:40.209 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:39 smithi173 ceph-mon[26787]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:40.209 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:39 smithi173 podman[51017]: 2024-05-15 12:38:39.806715174 +0000 UTC m=+0.016608893 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-05-15T12:38:40.209 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:39 smithi173 podman[51017]: 2024-05-15 12:38:39.924050886 +0000 UTC m=+0.133944583 container create 2e7edb6cbbab7299d2d23e9e141eff85ad86a63fc677c3d8dce98169982aeb1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., distribution-scope=public, name=ubi8, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, architecture=x86_64) 2024-05-15T12:38:40.210 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 podman[51017]: 2024-05-15 12:38:40.220450791 +0000 UTC m=+0.430344497 container init 2e7edb6cbbab7299d2d23e9e141eff85ad86a63fc677c3d8dce98169982aeb1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a, io.openshift.expose-services=, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vcs-type=git, io.buildah.version=1.32.0, version=8.5, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, distribution-scope=public) 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 podman[51017]: 2024-05-15 12:38:40.223518491 +0000 UTC m=+0.433412199 container start 2e7edb6cbbab7299d2d23e9e141eff85ad86a63fc677c3d8dce98169982aeb1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a, vendor=Red Hat, Inc., release=239.1651231664, version=8.5, name=ubi8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, maintainer=Paul Cuzner , io.openshift.tags=base rhel8) 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 bash[51017]: 2e7edb6cbbab7299d2d23e9e141eff85ad86a63fc677c3d8dce98169982aeb1c 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 systemd[1]: Started Ceph grafana.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374564252Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374782815Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374800354Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374807146Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374813225Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374819713Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374825275Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-05-15T12:38:40.461 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374831013Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374836952Z level=info msg="Path Home" path=/usr/share/grafana 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374842666Z level=info msg="Path Data" path=/var/lib/grafana 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374848112Z level=info msg="Path Logs" path=/var/log/grafana 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374853374Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374859337Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=settings t=2024-05-15T12:38:40.374864873Z level=info msg="App mode production" 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=sqlstore t=2024-05-15T12:38:40.374907899Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=sqlstore t=2024-05-15T12:38:40.374924229Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.375810192Z level=info msg="Starting DB migrations" 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.375893294Z level=info msg="Executing migration" id="create migration_log table" 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.376430194Z level=info msg="Executing migration" id="create user table" 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.376886831Z level=info msg="Executing migration" id="add unique index user.login" 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.377625401Z level=info msg="Executing migration" id="add unique index user.email" 2024-05-15T12:38:40.462 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.378473088Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.379428558Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.380195729Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.38102764Z level=info msg="Executing migration" id="create user table v2" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.381919262Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.382321912Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.385217821Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.38789394Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.39031547Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-05-15T12:38:40.463 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.393417457Z level=info msg="Executing migration" id="Update user table charset" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.396070533Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.396615139Z level=info msg="Executing migration" id="Add missing user data" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.399172292Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.399657542Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.401965088Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.402460935Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.404800933Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.407803865Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.410614598Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.413191628Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.416124785Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-05-15T12:38:40.464 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.418953645Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.421816821Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.42468423Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.427346248Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.429940854Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.432771031Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.435414764Z level=info msg="Executing migration" id="create temp_user v2" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.438113893Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.440856065Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.443345316Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.443778846Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.446665993Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.449288403Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.452020954Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-05-15T12:38:40.465 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.454768332Z level=info msg="Executing migration" id="create star table" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.457392889Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.460172102Z level=info msg="Executing migration" id="create org table v1" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.462834658Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.465721152Z level=info msg="Executing migration" id="create org_user table v1" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.466430007Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.467390212Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.468119962Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.469011792Z level=info msg="Executing migration" id="Update org table charset" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.469772886Z level=info msg="Executing migration" id="Update org_user table charset" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.47059078Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.471432364Z level=info msg="Executing migration" id="create dashboard table" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.472284905Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-05-15T12:38:40.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.473120282Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-05-15T12:38:40.711 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.473976658Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-05-15T12:38:40.711 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.474791212Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-05-15T12:38:40.711 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.475678085Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.476469352Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.477328131Z level=info msg="Executing migration" id="create dashboard v2" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.478119149Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.478939303Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.47997334Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.480655346Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.481518297Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.482423927Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.483316962Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.484038167Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.48491716Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-05-15T12:38:40.712 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.485763993Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.486623666Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.48747169Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.488334946Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.489224444Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.490180452Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.491003853Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.491966209Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.492675285Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.49358101Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.494448675Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.495422773Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-05-15T12:38:40.713 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.498178288Z level=info msg="Executing migration" id="Update dashboard title length" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.500850544Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.503671403Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.506493289Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.509282741Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.512272005Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.51491421Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.517668721Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.520346031Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.523308906Z level=info msg="Executing migration" id="Add check_sum column" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.526069234Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-05-15T12:38:40.714 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.52884023Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-05-15T12:38:40.716 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.531418673Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-05-15T12:38:40.716 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.534395188Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-05-15T12:38:40.716 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.537162138Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-05-15T12:38:40.716 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.539925324Z level=info msg="Executing migration" id="create data_source table" 2024-05-15T12:38:40.716 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.542732615Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-05-15T12:38:40.716 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.545807623Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-05-15T12:38:40.716 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.548368521Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.548932942Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.551545119Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.554048138Z level=info msg="Executing migration" id="create data_source table v2" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.556521232Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.559283611Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.561908167Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.56451153Z level=info msg="Executing migration" id="Add column with_credentials" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.56704489Z level=info msg="Executing migration" id="Add secure json data column" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.569872947Z level=info msg="Executing migration" id="Update data_source table charset" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.572279589Z level=info msg="Executing migration" id="Update initial version to 1" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.57264222Z level=info msg="Executing migration" id="Add read_only data column" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.573173053Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-05-15T12:38:40.717 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.573500418Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.573781484Z level=info msg="Executing migration" id="Add uid column" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.574317981Z level=info msg="Executing migration" id="Update uid value" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.574675304Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.575143708Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.575615516Z level=info msg="Executing migration" id="create api_key table" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.576040099Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.576804517Z level=info msg="Executing migration" id="add index api_key.key" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.577605539Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.578382411Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.578792661Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.579183282Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.579595934Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-05-15T12:38:40.718 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.580751575Z level=info msg="Executing migration" id="create api_key table v2" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.581169773Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.581632728Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.582047523Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.582477548Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.582834363Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.583164907Z level=info msg="Executing migration" id="Update api_key table charset" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.583470371Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.584004095Z level=info msg="Executing migration" id="Add service account foreign key" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.584596195Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.584931569Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.585509814Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.586102676Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-05-15T12:38:40.719 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.586522494Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.586912237Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.587300521Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.587753091Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.588192704Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.588651253Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.58892366Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.589188862Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.589796148Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.590391258Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.590725041Z level=info msg="Executing migration" id="create quota table v1" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.591137115Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.591605243Z level=info msg="Executing migration" id="Update quota table charset" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.591925977Z level=info msg="Executing migration" id="create plugin_setting table" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.592318552Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-05-15T12:38:40.720 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.592738863Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.593277109Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.593602165Z level=info msg="Executing migration" id="create session table" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.594044902Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.594371476Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.594704692Z level=info msg="Executing migration" id="create playlist table v2" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.595109142Z level=info msg="Executing migration" id="create playlist item table v2" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.595533501Z level=info msg="Executing migration" id="Update playlist table charset" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.59585087Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.596116433Z level=info msg="Executing migration" id="drop preferences table v2" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.596445361Z level=info msg="Executing migration" id="drop preferences table v3" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.596774484Z level=info msg="Executing migration" id="create preferences table v3" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.597188801Z level=info msg="Executing migration" id="Update preferences table charset" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.597557285Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-05-15T12:38:40.721 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.598149101Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.598475135Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.599104418Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.599738555Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.600062602Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.600529778Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.600992345Z level=info msg="Executing migration" id="create alert table v1" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.60177755Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.60256775Z level=info msg="Executing migration" id="add index alert state" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.603006744Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.603786753Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.604693717Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.605419849Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-05-15T12:38:40.722 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.605874333Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.607329293Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.608121465Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.609075591Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.609490238Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.60988549Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.610674673Z level=info msg="Executing migration" id="Add column is_default" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.611372634Z level=info msg="Executing migration" id="Add column frequency" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.612030006Z level=info msg="Executing migration" id="Add column send_reminder" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.612704817Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.613562972Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.614338941Z level=info msg="Executing migration" id="Update alert table charset" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.614665285Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.614936645Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-05-15T12:38:40.723 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.61532946Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.616230096Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.617111762Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.61785235Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.618280754Z level=info msg="Executing migration" id="Add for to alert table" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.61894981Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.61962881Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.620518899Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.621368727Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.621812832Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.624541481Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.62733678Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.629947859Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.632652486Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-05-15T12:38:40.724 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.635282872Z level=info msg="Executing migration" id="create annotation table v5" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.637882975Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.640795159Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.641253508Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.643855401Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.646627479Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.649435998Z level=info msg="Executing migration" id="Update annotation table charset" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.649834298Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.652493184Z level=info msg="Executing migration" id="Drop category_id index" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.655384589Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.658198859Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.658646065Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.661560696Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.664168562Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.66669545Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-05-15T12:38:40.725 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.667131706Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.670021859Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.672631241Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.675103727Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.675644286Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.678216617Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.678931573Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.680630447Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.683190421Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.683552446Z level=info msg="Executing migration" id="Add epoch_end column" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.686170552Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.686642746Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.688837315Z level=info msg="Executing migration" id="Move region to single row" 2024-05-15T12:38:40.726 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.689438925Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.691663988Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.69219036Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.692974992Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.693919962Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.694606751Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.695034523Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.695817694Z level=info msg="Executing migration" id="create test_data table" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.696562749Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.6969843Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.697769235Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.700526528Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.703299962Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.705794078Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-05-15T12:38:40.727 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.706122168Z level=info msg="Executing migration" id="create team table" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.70894953Z level=info msg="Executing migration" id="add index team.org_id" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.711694191Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.714283421Z level=info msg="Executing migration" id="create team member table" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.714703534Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.717155587Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.718121594Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.71882074Z level=info msg="Executing migration" id="Add column email to team table" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.719559703Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.720384847Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.721242561Z level=info msg="Executing migration" id="create dashboard acl table" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.722081408Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.722679543Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.723527582Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-05-15T12:38:40.728 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.72433371Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-05-15T12:38:40.729 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.725226422Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-05-15T12:38:40.729 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.725791189Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-05-15T12:38:40.729 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.726615353Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-05-15T12:38:40.961 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.727476896Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-05-15T12:38:40.961 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.728321517Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-05-15T12:38:40.961 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.729133348Z level=info msg="Executing migration" id="create tag table" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.729965404Z level=info msg="Executing migration" id="add index tag.key_value" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.730815244Z level=info msg="Executing migration" id="create login attempt table" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.731662162Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.7325127Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.733356864Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.736517231Z level=info msg="Executing migration" id="create login_attempt v2" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.737369884Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.738237968Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.739028315Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.739867565Z level=info msg="Executing migration" id="create user auth table" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.740713226Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.741560966Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-05-15T12:38:40.962 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.742487596Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.743419877Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.744277299Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.745118474Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.745982879Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.746816391Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.747642917Z level=info msg="Executing migration" id="create server_lock table" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.748674683Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.751391577Z level=info msg="Executing migration" id="create user auth token table" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.754389055Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.757111272Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.760063289Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.762945167Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-05-15T12:38:40.963 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.765616379Z level=info msg="Executing migration" id="create cache_data table" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.768508763Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.771009549Z level=info msg="Executing migration" id="create short_url table v1" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.773954001Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.776777686Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.779384669Z level=info msg="Executing migration" id="delete alert_definition table" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.782312063Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.785113204Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.788008725Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-05-15T12:38:40.964 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.790808344Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-05-15T12:38:40.965 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.793691594Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-05-15T12:38:40.965 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.796341712Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.799171774Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.801908494Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.804739965Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.807367039Z level=info msg="Executing migration" id="drop alert_definition table" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.810274075Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.812892598Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.815441225Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.818188002Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.820848525Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.823414922Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.826306267Z level=info msg="Executing migration" id="create alert_instance table" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.828850381Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-05-15T12:38:40.966 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.831587727Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.834320115Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.837147244Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.839726602Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.842465041Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.845961037Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.84933444Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.85017902Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.850997844Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.851851539Z level=info msg="Executing migration" id="create alert_rule table" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.85270492Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.853567194Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.854432359Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.855305828Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-05-15T12:38:40.967 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.856041184Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.856902194Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.857761868Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.858639023Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.859619657Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.860328343Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.861183035Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.86204058Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.862898035Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.863772164Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.864648821Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.865626962Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.866312516Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.867355003Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-05-15T12:38:40.968 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.868093975Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.86900001Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.870050646Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.870935747Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.87182468Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.872707685Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.873595685Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.874480025Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.875310375Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.876188643Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.877179431Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.878241463Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.879013769Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.881885061Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.884694053Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-05-15T12:38:40.969 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.887654451Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.888735755Z level=info msg="Executing migration" id="create provenance_type table" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.891510314Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.894391692Z level=info msg="Executing migration" id="create alert_image table" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.897042793Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.89986625Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.902662525Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.905431641Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.908087706Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.908247141Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.908712662Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.909521736Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.910354172Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.910617421Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-05-15T12:38:40.970 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.910831502Z level=info msg="alerts found to migrate" alerts=0 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.911426651Z level=info msg="Executing migration" id="create library_element table v1" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.912051854Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.912832373Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.913719211Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.914535353Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.915026277Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.91582221Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.916657564Z level=info msg="Executing migration" id="create data_keys table" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.917506253Z level=info msg="Executing migration" id="create secrets table" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.918092902Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.922693863Z level=info msg="Executing migration" id="add name column into data_keys" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.925621373Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.928300969Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-05-15T12:38:40.971 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.932865841Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.937678584Z level=info msg="Executing migration" id="create kv_store table v1" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.940183605Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.940731485Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.941086748Z level=info msg="Executing migration" id="create permission table" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.94152287Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.941956442Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.942429468Z level=info msg="Executing migration" id="create role table" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.942833763Z level=info msg="Executing migration" id="add column display_name" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.943781741Z level=info msg="Executing migration" id="add column group_name" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.945050678Z level=info msg="Executing migration" id="add index role.org_id" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.946001862Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.946475869Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.946923555Z level=info msg="Executing migration" id="create team role table" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.947319259Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-05-15T12:38:40.972 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.94776669Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.948234229Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.94867587Z level=info msg="Executing migration" id="create user role table" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.949061401Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.949508791Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.949948777Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.95040001Z level=info msg="Executing migration" id="create builtin role table" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.950855194Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.953556159Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.954246246Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.955323433Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.955821403Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.958560454Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.959043854Z level=info msg="Executing migration" id="add unique index role.uid" 2024-05-15T12:38:40.973 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.95961674Z level=info msg="Executing migration" id="create seed assignment table" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.960032041Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.960534772Z level=info msg="Executing migration" id="add column hidden to role table" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.961761554Z level=info msg="Executing migration" id="create query_history table v1" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.962283574Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.962747433Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.963037142Z level=info msg="Executing migration" id="teams permissions migration" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.963493234Z level=info msg="Executing migration" id="dashboard permissions" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.963981912Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.96438259Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.964713124Z level=info msg="Executing migration" id="alerting notification permissions" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.965067179Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.965491367Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.965944564Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-05-15T12:38:40.974 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.966966057Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.967294721Z level=info msg="Executing migration" id="create correlation table v1" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.967722056Z level=info msg="Executing migration" id="add index correlations.uid" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.968160718Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.968616377Z level=info msg="Executing migration" id="add correlation config column" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.970965066Z level=info msg="Executing migration" id="create entity_events table" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.971756492Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.972238392Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.972341403Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.972701732Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.972803541Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.973098154Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.973532045Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-05-15T12:38:40.975 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.975743079Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-05-15T12:38:41.245 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.978517033Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-05-15T12:38:41.245 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.979085051Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-05-15T12:38:41.245 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.981905237Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-05-15T12:38:41.245 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.98481367Z level=info msg="Executing migration" id="Drop public config table" 2024-05-15T12:38:41.245 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.987757835Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-05-15T12:38:41.245 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.988356754Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-05-15T12:38:41.245 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.990909581Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.9935103Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.996253819Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:40 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:40.999368029Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.00211578Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.004587623Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.007131075Z level=info msg="Executing migration" id="add share column" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.008159875Z level=info msg="Executing migration" id="create default alerting folders" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.010759508Z level=info msg="Executing migration" id="create file table" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.013483409Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.015901136Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-05-15T12:38:41.246 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.016397824Z level=info msg="Executing migration" id="create file_meta table" 2024-05-15T12:38:41.247 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.017235868Z level=info msg="Executing migration" id="file table idx: path key" 2024-05-15T12:38:41.247 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.019958907Z level=info msg="Executing migration" id="set path collation in file table" 2024-05-15T12:38:41.247 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.022571274Z level=info msg="Executing migration" id="managed permissions migration" 2024-05-15T12:38:41.247 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.022979364Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-05-15T12:38:41.247 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.025810569Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-05-15T12:38:41.247 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.026309012Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-05-15T12:38:41.247 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.028494173Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.02936413Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.0299607Z level=info msg="Executing migration" id="update group index for alert rules" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.030794601Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.031578317Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.032335825Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.033491248Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.034537342Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.035348565Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.045108986Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.045598676Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.046457843Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.049909514Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.050673521Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-05-15T12:38:41.248 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.051046614Z level=info msg="Executing migration" id="create folder table" 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.051824935Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.052681181Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.053492715Z level=info msg="Executing migration" id="Update folder title length" 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.053817128Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=migrator t=2024-05-15T12:38:41.054687449Z level=info msg="migrations completed" performed=485 skipped=0 duration=678.823912ms 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=sqlstore t=2024-05-15T12:38:41.05514799Z level=info msg="Created default organization" 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=plugin.loader t=2024-05-15T12:38:41.113613382Z level=info msg="Plugin registered" pluginID=input 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=plugin.loader t=2024-05-15T12:38:41.141093437Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=plugin.loader t=2024-05-15T12:38:41.141133735Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=secrets t=2024-05-15T12:38:41.141169344Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=query_data t=2024-05-15T12:38:41.144678761Z level=info msg="Query Service initialization" 2024-05-15T12:38:41.249 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=live.push_http t=2024-05-15T12:38:41.14724459Z level=info msg="Live Push Gateway initialization" 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[24799]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[24799]: Deploying daemon node-exporter.a on smithi102 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[31347]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:41.496 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.497 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:41 smithi102 ceph-mon[31347]: Deploying daemon node-exporter.a on smithi102 2024-05-15T12:38:41.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:41 smithi173 ceph-mon[26787]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:41.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:41 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.528 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:41 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:41 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:41 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:41.529 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:41 smithi173 ceph-mon[26787]: Deploying daemon node-exporter.a on smithi102 2024-05-15T12:38:41.902 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:41 smithi102 systemd[1]: Starting Ceph node-exporter.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:38:41.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:38:41.957 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-05-15T12:38:42.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=infra.usagestats.collector t=2024-05-15T12:38:41.793472486Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-05-15T12:38:42.056 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=provisioning.datasources t=2024-05-15T12:38:41.794713371Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=provisioning.datasources t=2024-05-15T12:38:41.806046435Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=provisioning.alerting t=2024-05-15T12:38:41.817083684Z level=info msg="starting to provision alerting" 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=provisioning.alerting t=2024-05-15T12:38:41.817103197Z level=info msg="finished to provision alerting" 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=grafanaStorageLogger t=2024-05-15T12:38:41.817314062Z level=info msg="storage starting" 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=ngalert.state.manager t=2024-05-15T12:38:41.8173951Z level=info msg="Warming state cache for startup" 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=http.server t=2024-05-15T12:38:41.819306944Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=ngalert.state.manager t=2024-05-15T12:38:41.828318772Z level=info msg="State cache has been initialized" states=0 duration=10.923037ms 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=ngalert.multiorg.alertmanager t=2024-05-15T12:38:41.828602321Z level=info msg="starting MultiOrg Alertmanager" 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=ticker t=2024-05-15T12:38:41.828626433Z level=info msg=starting first_tick=2024-05-15T12:38:50Z 2024-05-15T12:38:42.057 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:38:41 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a[51031]: logger=sqlstore.transactions t=2024-05-15T12:38:41.829931034Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-05-15T12:38:42.203 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:41 smithi102 bash[60793]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-05-15T12:38:42.488 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:42 smithi102 ceph-mon[24799]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:38:42.488 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:42 smithi102 ceph-mon[31347]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:38:42.691 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-05-15T12:38:40.340990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1062,"num_read_kb":905,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":228308,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749700140,"statfs":{"total":767926730752,"available":767692943360,"internally_reserved":0,"allocated":8187904,"data_stored":4433818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002578"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323082+0000","last_change":"2024-05-15T12:37:15.102600+0000","last_active":"2024-05-15T12:37:52.323082+0000","last_peered":"2024-05-15T12:37:52.323082+0000","last_clean":"2024-05-15T12:37:52.323082+0000","last_became_active":"2024-05-15T12:37:15.102241+0000","last_became_peered":"2024-05-15T12:37:15.102241+0000","last_unstale":"2024-05-15T12:37:52.323082+0000","last_undegraded":"2024-05-15T12:37:52.323082+0000","last_fullsized":"2024-05-15T12:37:52.323082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:06:20.143991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323627+0000","last_change":"2024-05-15T12:37:13.092914+0000","last_active":"2024-05-15T12:37:52.323627+0000","last_peered":"2024-05-15T12:37:52.323627+0000","last_clean":"2024-05-15T12:37:52.323627+0000","last_became_active":"2024-05-15T12:37:13.092398+0000","last_became_peered":"2024-05-15T12:37:13.092398+0000","last_unstale":"2024-05-15T12:37:52.323627+0000","last_undegraded":"2024-05-15T12:37:52.323627+0000","last_fullsized":"2024-05-15T12:37:52.323627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:17:08.171640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320990+0000","last_change":"2024-05-15T12:37:09.071444+0000","last_active":"2024-05-15T12:37:52.320990+0000","last_peered":"2024-05-15T12:37:52.320990+0000","last_clean":"2024-05-15T12:37:52.320990+0000","last_became_active":"2024-05-15T12:37:09.071161+0000","last_became_peered":"2024-05-15T12:37:09.071161+0000","last_unstale":"2024-05-15T12:37:52.320990+0000","last_undegraded":"2024-05-15T12:37:52.320990+0000","last_fullsized":"2024-05-15T12:37:52.320990+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:35:51.483393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.322873+0000","last_change":"2024-05-15T12:37:24.678381+0000","last_active":"2024-05-15T12:37:52.322873+0000","last_peered":"2024-05-15T12:37:52.322873+0000","last_clean":"2024-05-15T12:37:52.322873+0000","last_became_active":"2024-05-15T12:37:24.667945+0000","last_became_peered":"2024-05-15T12:37:24.667945+0000","last_unstale":"2024-05-15T12:37:52.322873+0000","last_undegraded":"2024-05-15T12:37:52.322873+0000","last_fullsized":"2024-05-15T12:37:52.322873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:33:49.381866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544670+0000","last_change":"2024-05-15T12:37:15.319547+0000","last_active":"2024-05-15T12:37:52.544670+0000","last_peered":"2024-05-15T12:37:52.544670+0000","last_clean":"2024-05-15T12:37:52.544670+0000","last_became_active":"2024-05-15T12:37:15.319041+0000","last_became_peered":"2024-05-15T12:37:15.319041+0000","last_unstale":"2024-05-15T12:37:52.544670+0000","last_undegraded":"2024-05-15T12:37:52.544670+0000","last_fullsized":"2024-05-15T12:37:52.544670+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:14:54.874371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.322907+0000","last_change":"2024-05-15T12:37:13.093049+0000","last_active":"2024-05-15T12:37:52.322907+0000","last_peered":"2024-05-15T12:37:52.322907+0000","last_clean":"2024-05-15T12:37:52.322907+0000","last_became_active":"2024-05-15T12:37:13.092704+0000","last_became_peered":"2024-05-15T12:37:13.092704+0000","last_unstale":"2024-05-15T12:37:52.322907+0000","last_undegraded":"2024-05-15T12:37:52.322907+0000","last_fullsized":"2024-05-15T12:37:52.322907+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:03:40.608770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545470+0000","last_change":"2024-05-15T12:37:09.065293+0000","last_active":"2024-05-15T12:37:52.545470+0000","last_peered":"2024-05-15T12:37:52.545470+0000","last_clean":"2024-05-15T12:37:52.545470+0000","last_became_active":"2024-05-15T12:37:09.065011+0000","last_became_peered":"2024-05-15T12:37:09.065011+0000","last_unstale":"2024-05-15T12:37:52.545470+0000","last_undegraded":"2024-05-15T12:37:52.545470+0000","last_fullsized":"2024-05-15T12:37:52.545470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:37:59.896309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544414+0000","last_change":"2024-05-15T12:37:11.084240+0000","last_active":"2024-05-15T12:37:52.544414+0000","last_peered":"2024-05-15T12:37:52.544414+0000","last_clean":"2024-05-15T12:37:52.544414+0000","last_became_active":"2024-05-15T12:37:11.083917+0000","last_became_peered":"2024-05-15T12:37:11.083917+0000","last_unstale":"2024-05-15T12:37:52.544414+0000","last_undegraded":"2024-05-15T12:37:52.544414+0000","last_fullsized":"2024-05-15T12:37:52.544414+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:21:16.143398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319786+0000","last_change":"2024-05-15T12:37:15.106034+0000","last_active":"2024-05-15T12:37:52.319786+0000","last_peered":"2024-05-15T12:37:52.319786+0000","last_clean":"2024-05-15T12:37:52.319786+0000","last_became_active":"2024-05-15T12:37:15.105669+0000","last_became_peered":"2024-05-15T12:37:15.105669+0000","last_unstale":"2024-05-15T12:37:52.319786+0000","last_undegraded":"2024-05-15T12:37:52.319786+0000","last_fullsized":"2024-05-15T12:37:52.319786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:36:23.541420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420635+0000","last_change":"2024-05-15T12:37:13.098122+0000","last_active":"2024-05-15T12:37:52.420635+0000","last_peered":"2024-05-15T12:37:52.420635+0000","last_clean":"2024-05-15T12:37:52.420635+0000","last_became_active":"2024-05-15T12:37:13.097726+0000","last_became_peered":"2024-05-15T12:37:13.097726+0000","last_unstale":"2024-05-15T12:37:52.420635+0000","last_undegraded":"2024-05-15T12:37:52.420635+0000","last_fullsized":"2024-05-15T12:37:52.420635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:23:01.705026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421017+0000","last_change":"2024-05-15T12:37:09.067399+0000","last_active":"2024-05-15T12:37:52.421017+0000","last_peered":"2024-05-15T12:37:52.421017+0000","last_clean":"2024-05-15T12:37:52.421017+0000","last_became_active":"2024-05-15T12:37:09.066761+0000","last_became_peered":"2024-05-15T12:37:09.066761+0000","last_unstale":"2024-05-15T12:37:52.421017+0000","last_undegraded":"2024-05-15T12:37:52.421017+0000","last_fullsized":"2024-05-15T12:37:52.421017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:16.154974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323252+0000","last_change":"2024-05-15T12:37:11.084043+0000","last_active":"2024-05-15T12:37:52.323252+0000","last_peered":"2024-05-15T12:37:52.323252+0000","last_clean":"2024-05-15T12:37:52.323252+0000","last_became_active":"2024-05-15T12:37:11.083710+0000","last_became_peered":"2024-05-15T12:37:11.083710+0000","last_unstale":"2024-05-15T12:37:52.323252+0000","last_undegraded":"2024-05-15T12:37:52.323252+0000","last_fullsized":"2024-05-15T12:37:52.323252+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:02:47.284700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420524+0000","last_change":"2024-05-15T12:37:15.103486+0000","last_active":"2024-05-15T12:37:52.420524+0000","last_peered":"2024-05-15T12:37:52.420524+0000","last_clean":"2024-05-15T12:37:52.420524+0000","last_became_active":"2024-05-15T12:37:15.103189+0000","last_became_peered":"2024-05-15T12:37:15.103189+0000","last_unstale":"2024-05-15T12:37:52.420524+0000","last_undegraded":"2024-05-15T12:37:52.420524+0000","last_fullsized":"2024-05-15T12:37:52.420524+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:59:24.916554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433597+0000","last_change":"2024-05-15T12:37:13.094897+0000","last_active":"2024-05-15T12:37:52.433597+0000","last_peered":"2024-05-15T12:37:52.433597+0000","last_clean":"2024-05-15T12:37:52.433597+0000","last_became_active":"2024-05-15T12:37:13.094582+0000","last_became_peered":"2024-05-15T12:37:13.094582+0000","last_unstale":"2024-05-15T12:37:52.433597+0000","last_undegraded":"2024-05-15T12:37:52.433597+0000","last_fullsized":"2024-05-15T12:37:52.433597+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:54:48.816395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420966+0000","last_change":"2024-05-15T12:37:09.070511+0000","last_active":"2024-05-15T12:37:52.420966+0000","last_peered":"2024-05-15T12:37:52.420966+0000","last_clean":"2024-05-15T12:37:52.420966+0000","last_became_active":"2024-05-15T12:37:09.070210+0000","last_became_peered":"2024-05-15T12:37:09.070210+0000","last_unstale":"2024-05-15T12:37:52.420966+0000","last_undegraded":"2024-05-15T12:37:52.420966+0000","last_fullsized":"2024-05-15T12:37:52.420966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:08:47.204599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323749+0000","last_change":"2024-05-15T12:37:11.087129+0000","last_active":"2024-05-15T12:37:52.323749+0000","last_peered":"2024-05-15T12:37:52.323749+0000","last_clean":"2024-05-15T12:37:52.323749+0000","last_became_active":"2024-05-15T12:37:11.086490+0000","last_became_peered":"2024-05-15T12:37:11.086490+0000","last_unstale":"2024-05-15T12:37:52.323749+0000","last_undegraded":"2024-05-15T12:37:52.323749+0000","last_fullsized":"2024-05-15T12:37:52.323749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:28:13.748140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324126+0000","last_change":"2024-05-15T12:37:15.316887+0000","last_active":"2024-05-15T12:37:52.324126+0000","last_peered":"2024-05-15T12:37:52.324126+0000","last_clean":"2024-05-15T12:37:52.324126+0000","last_became_active":"2024-05-15T12:37:15.316504+0000","last_became_peered":"2024-05-15T12:37:15.316504+0000","last_unstale":"2024-05-15T12:37:52.324126+0000","last_undegraded":"2024-05-15T12:37:52.324126+0000","last_fullsized":"2024-05-15T12:37:52.324126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:49:58.474167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324615+0000","last_change":"2024-05-15T12:37:13.083240+0000","last_active":"2024-05-15T12:37:52.324615+0000","last_peered":"2024-05-15T12:37:52.324615+0000","last_clean":"2024-05-15T12:37:52.324615+0000","last_became_active":"2024-05-15T12:37:13.082899+0000","last_became_peered":"2024-05-15T12:37:13.082899+0000","last_unstale":"2024-05-15T12:37:52.324615+0000","last_undegraded":"2024-05-15T12:37:52.324615+0000","last_fullsized":"2024-05-15T12:37:52.324615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:58:36.819505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320145+0000","last_change":"2024-05-15T12:37:09.069604+0000","last_active":"2024-05-15T12:37:52.320145+0000","last_peered":"2024-05-15T12:37:52.320145+0000","last_clean":"2024-05-15T12:37:52.320145+0000","last_became_active":"2024-05-15T12:37:09.068379+0000","last_became_peered":"2024-05-15T12:37:09.068379+0000","last_unstale":"2024-05-15T12:37:52.320145+0000","last_undegraded":"2024-05-15T12:37:52.320145+0000","last_fullsized":"2024-05-15T12:37:52.320145+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:17:03.808516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428962+0000","last_change":"2024-05-15T12:37:11.082781+0000","last_active":"2024-05-15T12:37:52.428962+0000","last_peered":"2024-05-15T12:37:52.428962+0000","last_clean":"2024-05-15T12:37:52.428962+0000","last_became_active":"2024-05-15T12:37:11.082497+0000","last_became_peered":"2024-05-15T12:37:11.082497+0000","last_unstale":"2024-05-15T12:37:52.428962+0000","last_undegraded":"2024-05-15T12:37:52.428962+0000","last_fullsized":"2024-05-15T12:37:52.428962+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:33:41.512925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545832+0000","last_change":"2024-05-15T12:37:15.316767+0000","last_active":"2024-05-15T12:37:52.545832+0000","last_peered":"2024-05-15T12:37:52.545832+0000","last_clean":"2024-05-15T12:37:52.545832+0000","last_became_active":"2024-05-15T12:37:15.316340+0000","last_became_peered":"2024-05-15T12:37:15.316340+0000","last_unstale":"2024-05-15T12:37:52.545832+0000","last_undegraded":"2024-05-15T12:37:52.545832+0000","last_fullsized":"2024-05-15T12:37:52.545832+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:44:14.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323529+0000","last_change":"2024-05-15T12:37:13.092984+0000","last_active":"2024-05-15T12:37:52.323529+0000","last_peered":"2024-05-15T12:37:52.323529+0000","last_clean":"2024-05-15T12:37:52.323529+0000","last_became_active":"2024-05-15T12:37:13.092666+0000","last_became_peered":"2024-05-15T12:37:13.092666+0000","last_unstale":"2024-05-15T12:37:52.323529+0000","last_undegraded":"2024-05-15T12:37:52.323529+0000","last_fullsized":"2024-05-15T12:37:52.323529+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:26:13.925238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323327+0000","last_change":"2024-05-15T12:37:09.066768+0000","last_active":"2024-05-15T12:37:52.323327+0000","last_peered":"2024-05-15T12:37:52.323327+0000","last_clean":"2024-05-15T12:37:52.323327+0000","last_became_active":"2024-05-15T12:37:09.066431+0000","last_became_peered":"2024-05-15T12:37:09.066431+0000","last_unstale":"2024-05-15T12:37:52.323327+0000","last_undegraded":"2024-05-15T12:37:52.323327+0000","last_fullsized":"2024-05-15T12:37:52.323327+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:21:07.246383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545013+0000","last_change":"2024-05-15T12:37:11.087059+0000","last_active":"2024-05-15T12:37:52.545013+0000","last_peered":"2024-05-15T12:37:52.545013+0000","last_clean":"2024-05-15T12:37:52.545013+0000","last_became_active":"2024-05-15T12:37:11.086355+0000","last_became_peered":"2024-05-15T12:37:11.086355+0000","last_unstale":"2024-05-15T12:37:52.545013+0000","last_undegraded":"2024-05-15T12:37:52.545013+0000","last_fullsized":"2024-05-15T12:37:52.545013+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:18:52.297504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433929+0000","last_change":"2024-05-15T12:37:15.107738+0000","last_active":"2024-05-15T12:37:52.433929+0000","last_peered":"2024-05-15T12:37:52.433929+0000","last_clean":"2024-05-15T12:37:52.433929+0000","last_became_active":"2024-05-15T12:37:15.107357+0000","last_became_peered":"2024-05-15T12:37:15.107357+0000","last_unstale":"2024-05-15T12:37:52.433929+0000","last_undegraded":"2024-05-15T12:37:52.433929+0000","last_fullsized":"2024-05-15T12:37:52.433929+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:34:22.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'11","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129400+0000","last_change":"2024-05-15T12:37:13.095415+0000","last_active":"2024-05-15T12:38:29.129400+0000","last_peered":"2024-05-15T12:38:29.129400+0000","last_clean":"2024-05-15T12:38:29.129400+0000","last_became_active":"2024-05-15T12:37:13.095086+0000","last_became_peered":"2024-05-15T12:37:13.095086+0000","last_unstale":"2024-05-15T12:38:29.129400+0000","last_undegraded":"2024-05-15T12:38:29.129400+0000","last_fullsized":"2024-05-15T12:38:29.129400+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:57:59.443236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323718+0000","last_change":"2024-05-15T12:37:09.069433+0000","last_active":"2024-05-15T12:37:52.323718+0000","last_peered":"2024-05-15T12:37:52.323718+0000","last_clean":"2024-05-15T12:37:52.323718+0000","last_became_active":"2024-05-15T12:37:09.068957+0000","last_became_peered":"2024-05-15T12:37:09.068957+0000","last_unstale":"2024-05-15T12:37:52.323718+0000","last_undegraded":"2024-05-15T12:37:52.323718+0000","last_fullsized":"2024-05-15T12:37:52.323718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:20:17.612056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320928+0000","last_change":"2024-05-15T12:37:11.080224+0000","last_active":"2024-05-15T12:37:52.320928+0000","last_peered":"2024-05-15T12:37:52.320928+0000","last_clean":"2024-05-15T12:37:52.320928+0000","last_became_active":"2024-05-15T12:37:11.079961+0000","last_became_peered":"2024-05-15T12:37:11.079961+0000","last_unstale":"2024-05-15T12:37:52.320928+0000","last_undegraded":"2024-05-15T12:37:52.320928+0000","last_fullsized":"2024-05-15T12:37:52.320928+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:56:54.697203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324755+0000","last_change":"2024-05-15T12:37:15.102480+0000","last_active":"2024-05-15T12:37:52.324755+0000","last_peered":"2024-05-15T12:37:52.324755+0000","last_clean":"2024-05-15T12:37:52.324755+0000","last_became_active":"2024-05-15T12:37:15.102154+0000","last_became_peered":"2024-05-15T12:37:15.102154+0000","last_unstale":"2024-05-15T12:37:52.324755+0000","last_undegraded":"2024-05-15T12:37:52.324755+0000","last_fullsized":"2024-05-15T12:37:52.324755+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:31:30.098668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.321070+0000","last_change":"2024-05-15T12:37:13.089935+0000","last_active":"2024-05-15T12:37:52.321070+0000","last_peered":"2024-05-15T12:37:52.321070+0000","last_clean":"2024-05-15T12:37:52.321070+0000","last_became_active":"2024-05-15T12:37:13.089530+0000","last_became_peered":"2024-05-15T12:37:13.089530+0000","last_unstale":"2024-05-15T12:37:52.321070+0000","last_undegraded":"2024-05-15T12:37:52.321070+0000","last_fullsized":"2024-05-15T12:37:52.321070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:19:19.178879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544154+0000","last_change":"2024-05-15T12:37:09.073841+0000","last_active":"2024-05-15T12:37:52.544154+0000","last_peered":"2024-05-15T12:37:52.544154+0000","last_clean":"2024-05-15T12:37:52.544154+0000","last_became_active":"2024-05-15T12:37:09.073549+0000","last_became_peered":"2024-05-15T12:37:09.073549+0000","last_unstale":"2024-05-15T12:37:52.544154+0000","last_undegraded":"2024-05-15T12:37:52.544154+0000","last_fullsized":"2024-05-15T12:37:52.544154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:31:02.467093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544863+0000","last_change":"2024-05-15T12:37:11.085950+0000","last_active":"2024-05-15T12:37:52.544863+0000","last_peered":"2024-05-15T12:37:52.544863+0000","last_clean":"2024-05-15T12:37:52.544863+0000","last_became_active":"2024-05-15T12:37:11.085631+0000","last_became_peered":"2024-05-15T12:37:11.085631+0000","last_unstale":"2024-05-15T12:37:52.544863+0000","last_undegraded":"2024-05-15T12:37:52.544863+0000","last_fullsized":"2024-05-15T12:37:52.544863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:28:52.396142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433582+0000","last_change":"2024-05-15T12:37:15.106738+0000","last_active":"2024-05-15T12:37:52.433582+0000","last_peered":"2024-05-15T12:37:52.433582+0000","last_clean":"2024-05-15T12:37:52.433582+0000","last_became_active":"2024-05-15T12:37:15.106417+0000","last_became_peered":"2024-05-15T12:37:15.106417+0000","last_unstale":"2024-05-15T12:37:52.433582+0000","last_undegraded":"2024-05-15T12:37:52.433582+0000","last_fullsized":"2024-05-15T12:37:52.433582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:37:20.699673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544251+0000","last_change":"2024-05-15T12:37:13.093826+0000","last_active":"2024-05-15T12:37:52.544251+0000","last_peered":"2024-05-15T12:37:52.544251+0000","last_clean":"2024-05-15T12:37:52.544251+0000","last_became_active":"2024-05-15T12:37:13.093542+0000","last_became_peered":"2024-05-15T12:37:13.093542+0000","last_unstale":"2024-05-15T12:37:52.544251+0000","last_undegraded":"2024-05-15T12:37:52.544251+0000","last_fullsized":"2024-05-15T12:37:52.544251+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:11:10.979708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.321043+0000","last_change":"2024-05-15T12:37:09.070640+0000","last_active":"2024-05-15T12:37:52.321043+0000","last_peered":"2024-05-15T12:37:52.321043+0000","last_clean":"2024-05-15T12:37:52.321043+0000","last_became_active":"2024-05-15T12:37:09.070372+0000","last_became_peered":"2024-05-15T12:37:09.070372+0000","last_unstale":"2024-05-15T12:37:52.321043+0000","last_undegraded":"2024-05-15T12:37:52.321043+0000","last_fullsized":"2024-05-15T12:37:52.321043+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:59:46.603095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544165+0000","last_change":"2024-05-15T12:37:11.088186+0000","last_active":"2024-05-15T12:37:52.544165+0000","last_peered":"2024-05-15T12:37:52.544165+0000","last_clean":"2024-05-15T12:37:52.544165+0000","last_became_active":"2024-05-15T12:37:11.087828+0000","last_became_peered":"2024-05-15T12:37:11.087828+0000","last_unstale":"2024-05-15T12:37:52.544165+0000","last_undegraded":"2024-05-15T12:37:52.544165+0000","last_fullsized":"2024-05-15T12:37:52.544165+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:09:53.788757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544470+0000","last_change":"2024-05-15T12:37:15.319273+0000","last_active":"2024-05-15T12:37:52.544470+0000","last_peered":"2024-05-15T12:37:52.544470+0000","last_clean":"2024-05-15T12:37:52.544470+0000","last_became_active":"2024-05-15T12:37:15.318887+0000","last_became_peered":"2024-05-15T12:37:15.318887+0000","last_unstale":"2024-05-15T12:37:52.544470+0000","last_undegraded":"2024-05-15T12:37:52.544470+0000","last_fullsized":"2024-05-15T12:37:52.544470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:00:51.121558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433718+0000","last_change":"2024-05-15T12:37:13.095074+0000","last_active":"2024-05-15T12:37:52.433718+0000","last_peered":"2024-05-15T12:37:52.433718+0000","last_clean":"2024-05-15T12:37:52.433718+0000","last_became_active":"2024-05-15T12:37:13.094778+0000","last_became_peered":"2024-05-15T12:37:13.094778+0000","last_unstale":"2024-05-15T12:37:52.433718+0000","last_undegraded":"2024-05-15T12:37:52.433718+0000","last_fullsized":"2024-05-15T12:37:52.433718+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:26:08.311809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421017+0000","last_change":"2024-05-15T12:37:09.071500+0000","last_active":"2024-05-15T12:37:52.421017+0000","last_peered":"2024-05-15T12:37:52.421017+0000","last_clean":"2024-05-15T12:37:52.421017+0000","last_became_active":"2024-05-15T12:37:09.071211+0000","last_became_peered":"2024-05-15T12:37:09.071211+0000","last_unstale":"2024-05-15T12:37:52.421017+0000","last_undegraded":"2024-05-15T12:37:52.421017+0000","last_fullsized":"2024-05-15T12:37:52.421017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:02:16.995090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544384+0000","last_change":"2024-05-15T12:37:11.084426+0000","last_active":"2024-05-15T12:37:52.544384+0000","last_peered":"2024-05-15T12:37:52.544384+0000","last_clean":"2024-05-15T12:37:52.544384+0000","last_became_active":"2024-05-15T12:37:11.084110+0000","last_became_peered":"2024-05-15T12:37:11.084110+0000","last_unstale":"2024-05-15T12:37:52.544384+0000","last_undegraded":"2024-05-15T12:37:52.544384+0000","last_fullsized":"2024-05-15T12:37:52.544384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:58:54.713264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319498+0000","last_change":"2024-05-15T12:37:15.105516+0000","last_active":"2024-05-15T12:37:52.319498+0000","last_peered":"2024-05-15T12:37:52.319498+0000","last_clean":"2024-05-15T12:37:52.319498+0000","last_became_active":"2024-05-15T12:37:15.105201+0000","last_became_peered":"2024-05-15T12:37:15.105201+0000","last_unstale":"2024-05-15T12:37:52.319498+0000","last_undegraded":"2024-05-15T12:37:52.319498+0000","last_fullsized":"2024-05-15T12:37:52.319498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:12:33.897811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544763+0000","last_change":"2024-05-15T12:37:13.090467+0000","last_active":"2024-05-15T12:37:52.544763+0000","last_peered":"2024-05-15T12:37:52.544763+0000","last_clean":"2024-05-15T12:37:52.544763+0000","last_became_active":"2024-05-15T12:37:13.090091+0000","last_became_peered":"2024-05-15T12:37:13.090091+0000","last_unstale":"2024-05-15T12:37:52.544763+0000","last_undegraded":"2024-05-15T12:37:52.544763+0000","last_fullsized":"2024-05-15T12:37:52.544763+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:21:25.235934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433998+0000","last_change":"2024-05-15T12:37:09.072613+0000","last_active":"2024-05-15T12:37:52.433998+0000","last_peered":"2024-05-15T12:37:52.433998+0000","last_clean":"2024-05-15T12:37:52.433998+0000","last_became_active":"2024-05-15T12:37:09.072271+0000","last_became_peered":"2024-05-15T12:37:09.072271+0000","last_unstale":"2024-05-15T12:37:52.433998+0000","last_undegraded":"2024-05-15T12:37:52.433998+0000","last_fullsized":"2024-05-15T12:37:52.433998+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:11:26.173203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324100+0000","last_change":"2024-05-15T12:37:11.081801+0000","last_active":"2024-05-15T12:37:52.324100+0000","last_peered":"2024-05-15T12:37:52.324100+0000","last_clean":"2024-05-15T12:37:52.324100+0000","last_became_active":"2024-05-15T12:37:11.080039+0000","last_became_peered":"2024-05-15T12:37:11.080039+0000","last_unstale":"2024-05-15T12:37:52.324100+0000","last_undegraded":"2024-05-15T12:37:52.324100+0000","last_fullsized":"2024-05-15T12:37:52.324100+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:54:56.958666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544154+0000","last_change":"2024-05-15T12:37:15.103262+0000","last_active":"2024-05-15T12:37:52.544154+0000","last_peered":"2024-05-15T12:37:52.544154+0000","last_clean":"2024-05-15T12:37:52.544154+0000","last_became_active":"2024-05-15T12:37:15.102965+0000","last_became_peered":"2024-05-15T12:37:15.102965+0000","last_unstale":"2024-05-15T12:37:52.544154+0000","last_undegraded":"2024-05-15T12:37:52.544154+0000","last_fullsized":"2024-05-15T12:37:52.544154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:31:24.103174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324003+0000","last_change":"2024-05-15T12:37:13.093157+0000","last_active":"2024-05-15T12:37:52.324003+0000","last_peered":"2024-05-15T12:37:52.324003+0000","last_clean":"2024-05-15T12:37:52.324003+0000","last_became_active":"2024-05-15T12:37:13.092802+0000","last_became_peered":"2024-05-15T12:37:13.092802+0000","last_unstale":"2024-05-15T12:37:52.324003+0000","last_undegraded":"2024-05-15T12:37:52.324003+0000","last_fullsized":"2024-05-15T12:37:52.324003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:10:42.928759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323851+0000","last_change":"2024-05-15T12:37:09.072446+0000","last_active":"2024-05-15T12:37:52.323851+0000","last_peered":"2024-05-15T12:37:52.323851+0000","last_clean":"2024-05-15T12:37:52.323851+0000","last_became_active":"2024-05-15T12:37:09.072109+0000","last_became_peered":"2024-05-15T12:37:09.072109+0000","last_unstale":"2024-05-15T12:37:52.323851+0000","last_undegraded":"2024-05-15T12:37:52.323851+0000","last_fullsized":"2024-05-15T12:37:52.323851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:30:07.904230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323968+0000","last_change":"2024-05-15T12:37:24.684617+0000","last_active":"2024-05-15T12:37:52.323968+0000","last_peered":"2024-05-15T12:37:52.323968+0000","last_clean":"2024-05-15T12:37:52.323968+0000","last_became_active":"2024-05-15T12:37:24.668904+0000","last_became_peered":"2024-05-15T12:37:24.668904+0000","last_unstale":"2024-05-15T12:37:52.323968+0000","last_undegraded":"2024-05-15T12:37:52.323968+0000","last_fullsized":"2024-05-15T12:37:52.323968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:14:37.439983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320676+0000","last_change":"2024-05-15T12:37:15.101675+0000","last_active":"2024-05-15T12:37:52.320676+0000","last_peered":"2024-05-15T12:37:52.320676+0000","last_clean":"2024-05-15T12:37:52.320676+0000","last_became_active":"2024-05-15T12:37:15.101362+0000","last_became_peered":"2024-05-15T12:37:15.101362+0000","last_unstale":"2024-05-15T12:37:52.320676+0000","last_undegraded":"2024-05-15T12:37:52.320676+0000","last_fullsized":"2024-05-15T12:37:52.320676+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:43:18.765531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129384+0000","last_change":"2024-05-15T12:37:13.095655+0000","last_active":"2024-05-15T12:38:29.129384+0000","last_peered":"2024-05-15T12:38:29.129384+0000","last_clean":"2024-05-15T12:38:29.129384+0000","last_became_active":"2024-05-15T12:37:13.095361+0000","last_became_peered":"2024-05-15T12:37:13.095361+0000","last_unstale":"2024-05-15T12:38:29.129384+0000","last_undegraded":"2024-05-15T12:38:29.129384+0000","last_fullsized":"2024-05-15T12:38:29.129384+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:47:10.369343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433787+0000","last_change":"2024-05-15T12:37:09.072627+0000","last_active":"2024-05-15T12:37:52.433787+0000","last_peered":"2024-05-15T12:37:52.433787+0000","last_clean":"2024-05-15T12:37:52.433787+0000","last_became_active":"2024-05-15T12:37:09.072308+0000","last_became_peered":"2024-05-15T12:37:09.072308+0000","last_unstale":"2024-05-15T12:37:52.433787+0000","last_undegraded":"2024-05-15T12:37:52.433787+0000","last_fullsized":"2024-05-15T12:37:52.433787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:26:24.723885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545054+0000","last_change":"2024-05-15T12:37:11.089554+0000","last_active":"2024-05-15T12:37:52.545054+0000","last_peered":"2024-05-15T12:37:52.545054+0000","last_clean":"2024-05-15T12:37:52.545054+0000","last_became_active":"2024-05-15T12:37:11.088650+0000","last_became_peered":"2024-05-15T12:37:11.088650+0000","last_unstale":"2024-05-15T12:37:52.545054+0000","last_undegraded":"2024-05-15T12:37:52.545054+0000","last_fullsized":"2024-05-15T12:37:52.545054+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:18:02.844729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324342+0000","last_change":"2024-05-15T12:37:15.103999+0000","last_active":"2024-05-15T12:37:52.324342+0000","last_peered":"2024-05-15T12:37:52.324342+0000","last_clean":"2024-05-15T12:37:52.324342+0000","last_became_active":"2024-05-15T12:37:15.103613+0000","last_became_peered":"2024-05-15T12:37:15.103613+0000","last_unstale":"2024-05-15T12:37:52.324342+0000","last_undegraded":"2024-05-15T12:37:52.324342+0000","last_fullsized":"2024-05-15T12:37:52.324342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:12:04.291749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129297+0000","last_change":"2024-05-15T12:37:13.092841+0000","last_active":"2024-05-15T12:38:29.129297+0000","last_peered":"2024-05-15T12:38:29.129297+0000","last_clean":"2024-05-15T12:38:29.129297+0000","last_became_active":"2024-05-15T12:37:13.092552+0000","last_became_peered":"2024-05-15T12:37:13.092552+0000","last_unstale":"2024-05-15T12:38:29.129297+0000","last_undegraded":"2024-05-15T12:38:29.129297+0000","last_fullsized":"2024-05-15T12:38:29.129297+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:39:13.880161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320046+0000","last_change":"2024-05-15T12:37:09.067840+0000","last_active":"2024-05-15T12:37:52.320046+0000","last_peered":"2024-05-15T12:37:52.320046+0000","last_clean":"2024-05-15T12:37:52.320046+0000","last_became_active":"2024-05-15T12:37:09.067100+0000","last_became_peered":"2024-05-15T12:37:09.067100+0000","last_unstale":"2024-05-15T12:37:52.320046+0000","last_undegraded":"2024-05-15T12:37:52.320046+0000","last_fullsized":"2024-05-15T12:37:52.320046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:16:07.291371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421185+0000","last_change":"2024-05-15T12:37:11.084780+0000","last_active":"2024-05-15T12:37:52.421185+0000","last_peered":"2024-05-15T12:37:52.421185+0000","last_clean":"2024-05-15T12:37:52.421185+0000","last_became_active":"2024-05-15T12:37:11.084439+0000","last_became_peered":"2024-05-15T12:37:11.084439+0000","last_unstale":"2024-05-15T12:37:52.421185+0000","last_undegraded":"2024-05-15T12:37:52.421185+0000","last_fullsized":"2024-05-15T12:37:52.421185+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:37.629599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.429198+0000","last_change":"2024-05-15T12:37:15.107416+0000","last_active":"2024-05-15T12:37:52.429198+0000","last_peered":"2024-05-15T12:37:52.429198+0000","last_clean":"2024-05-15T12:37:52.429198+0000","last_became_active":"2024-05-15T12:37:15.107084+0000","last_became_peered":"2024-05-15T12:37:15.107084+0000","last_unstale":"2024-05-15T12:37:52.429198+0000","last_undegraded":"2024-05-15T12:37:52.429198+0000","last_fullsized":"2024-05-15T12:37:52.429198+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:23:27.889778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545779+0000","last_change":"2024-05-15T12:37:13.091053+0000","last_active":"2024-05-15T12:37:52.545779+0000","last_peered":"2024-05-15T12:37:52.545779+0000","last_clean":"2024-05-15T12:37:52.545779+0000","last_became_active":"2024-05-15T12:37:13.090720+0000","last_became_peered":"2024-05-15T12:37:13.090720+0000","last_unstale":"2024-05-15T12:37:52.545779+0000","last_undegraded":"2024-05-15T12:37:52.545779+0000","last_fullsized":"2024-05-15T12:37:52.545779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:36:48.582844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'1","reported_seq":41,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433569+0000","last_change":"2024-05-15T12:37:09.073488+0000","last_active":"2024-05-15T12:37:52.433569+0000","last_peered":"2024-05-15T12:37:52.433569+0000","last_clean":"2024-05-15T12:37:52.433569+0000","last_became_active":"2024-05-15T12:37:09.073181+0000","last_became_peered":"2024-05-15T12:37:09.073181+0000","last_unstale":"2024-05-15T12:37:52.433569+0000","last_undegraded":"2024-05-15T12:37:52.433569+0000","last_fullsized":"2024-05-15T12:37:52.433569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:32:47.183558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320933+0000","last_change":"2024-05-15T12:37:11.080880+0000","last_active":"2024-05-15T12:37:52.320933+0000","last_peered":"2024-05-15T12:37:52.320933+0000","last_clean":"2024-05-15T12:37:52.320933+0000","last_became_active":"2024-05-15T12:37:11.080611+0000","last_became_peered":"2024-05-15T12:37:11.080611+0000","last_unstale":"2024-05-15T12:37:52.320933+0000","last_undegraded":"2024-05-15T12:37:52.320933+0000","last_fullsized":"2024-05-15T12:37:52.320933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:39:59.310154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433982+0000","last_change":"2024-05-15T12:37:15.318629+0000","last_active":"2024-05-15T12:37:52.433982+0000","last_peered":"2024-05-15T12:37:52.433982+0000","last_clean":"2024-05-15T12:37:52.433982+0000","last_became_active":"2024-05-15T12:37:15.318289+0000","last_became_peered":"2024-05-15T12:37:15.318289+0000","last_unstale":"2024-05-15T12:37:52.433982+0000","last_undegraded":"2024-05-15T12:37:52.433982+0000","last_fullsized":"2024-05-15T12:37:52.433982+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:41:50.527836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421051+0000","last_change":"2024-05-15T12:37:13.096162+0000","last_active":"2024-05-15T12:37:52.421051+0000","last_peered":"2024-05-15T12:37:52.421051+0000","last_clean":"2024-05-15T12:37:52.421051+0000","last_became_active":"2024-05-15T12:37:13.095884+0000","last_became_peered":"2024-05-15T12:37:13.095884+0000","last_unstale":"2024-05-15T12:37:52.421051+0000","last_undegraded":"2024-05-15T12:37:52.421051+0000","last_fullsized":"2024-05-15T12:37:52.421051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:27:10.590238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544243+0000","last_change":"2024-05-15T12:37:09.068210+0000","last_active":"2024-05-15T12:37:52.544243+0000","last_peered":"2024-05-15T12:37:52.544243+0000","last_clean":"2024-05-15T12:37:52.544243+0000","last_became_active":"2024-05-15T12:37:09.067827+0000","last_became_peered":"2024-05-15T12:37:09.067827+0000","last_unstale":"2024-05-15T12:37:52.544243+0000","last_undegraded":"2024-05-15T12:37:52.544243+0000","last_fullsized":"2024-05-15T12:37:52.544243+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:30:55.087299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545877+0000","last_change":"2024-05-15T12:37:11.089151+0000","last_active":"2024-05-15T12:37:52.545877+0000","last_peered":"2024-05-15T12:37:52.545877+0000","last_clean":"2024-05-15T12:37:52.545877+0000","last_became_active":"2024-05-15T12:37:11.088858+0000","last_became_peered":"2024-05-15T12:37:11.088858+0000","last_unstale":"2024-05-15T12:37:52.545877+0000","last_undegraded":"2024-05-15T12:37:52.545877+0000","last_fullsized":"2024-05-15T12:37:52.545877+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:02:51.858503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420784+0000","last_change":"2024-05-15T12:37:15.316724+0000","last_active":"2024-05-15T12:37:52.420784+0000","last_peered":"2024-05-15T12:37:52.420784+0000","last_clean":"2024-05-15T12:37:52.420784+0000","last_became_active":"2024-05-15T12:37:15.316391+0000","last_became_peered":"2024-05-15T12:37:15.316391+0000","last_unstale":"2024-05-15T12:37:52.420784+0000","last_undegraded":"2024-05-15T12:37:52.420784+0000","last_fullsized":"2024-05-15T12:37:52.420784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:57.229038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434345+0000","last_change":"2024-05-15T12:37:09.074136+0000","last_active":"2024-05-15T12:37:52.434345+0000","last_peered":"2024-05-15T12:37:52.434345+0000","last_clean":"2024-05-15T12:37:52.434345+0000","last_became_active":"2024-05-15T12:37:09.073851+0000","last_became_peered":"2024-05-15T12:37:09.073851+0000","last_unstale":"2024-05-15T12:37:52.434345+0000","last_undegraded":"2024-05-15T12:37:52.434345+0000","last_fullsized":"2024-05-15T12:37:52.434345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:29:57.880157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420550+0000","last_change":"2024-05-15T12:37:11.084555+0000","last_active":"2024-05-15T12:37:52.420550+0000","last_peered":"2024-05-15T12:37:52.420550+0000","last_clean":"2024-05-15T12:37:52.420550+0000","last_became_active":"2024-05-15T12:37:11.084182+0000","last_became_peered":"2024-05-15T12:37:11.084182+0000","last_unstale":"2024-05-15T12:37:52.420550+0000","last_undegraded":"2024-05-15T12:37:52.420550+0000","last_fullsized":"2024-05-15T12:37:52.420550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:07:24.527454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129387+0000","last_change":"2024-05-15T12:37:13.094232+0000","last_active":"2024-05-15T12:38:29.129387+0000","last_peered":"2024-05-15T12:38:29.129387+0000","last_clean":"2024-05-15T12:38:29.129387+0000","last_became_active":"2024-05-15T12:37:13.093938+0000","last_became_peered":"2024-05-15T12:37:13.093938+0000","last_unstale":"2024-05-15T12:38:29.129387+0000","last_undegraded":"2024-05-15T12:38:29.129387+0000","last_fullsized":"2024-05-15T12:38:29.129387+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:01:05.717989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544574+0000","last_change":"2024-05-15T12:37:15.109119+0000","last_active":"2024-05-15T12:37:52.544574+0000","last_peered":"2024-05-15T12:37:52.544574+0000","last_clean":"2024-05-15T12:37:52.544574+0000","last_became_active":"2024-05-15T12:37:15.108741+0000","last_became_peered":"2024-05-15T12:37:15.108741+0000","last_unstale":"2024-05-15T12:37:52.544574+0000","last_undegraded":"2024-05-15T12:37:52.544574+0000","last_fullsized":"2024-05-15T12:37:52.544574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:57:18.964707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434185+0000","last_change":"2024-05-15T12:37:09.072892+0000","last_active":"2024-05-15T12:37:52.434185+0000","last_peered":"2024-05-15T12:37:52.434185+0000","last_clean":"2024-05-15T12:37:52.434185+0000","last_became_active":"2024-05-15T12:37:09.072354+0000","last_became_peered":"2024-05-15T12:37:09.072354+0000","last_unstale":"2024-05-15T12:37:52.434185+0000","last_undegraded":"2024-05-15T12:37:52.434185+0000","last_fullsized":"2024-05-15T12:37:52.434185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:35:29.903053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323305+0000","last_change":"2024-05-15T12:37:11.092438+0000","last_active":"2024-05-15T12:37:52.323305+0000","last_peered":"2024-05-15T12:37:52.323305+0000","last_clean":"2024-05-15T12:37:52.323305+0000","last_became_active":"2024-05-15T12:37:11.092120+0000","last_became_peered":"2024-05-15T12:37:11.092120+0000","last_unstale":"2024-05-15T12:37:52.323305+0000","last_undegraded":"2024-05-15T12:37:52.323305+0000","last_fullsized":"2024-05-15T12:37:52.323305+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:16:27.383512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428364+0000","last_change":"2024-05-15T12:37:13.095561+0000","last_active":"2024-05-15T12:37:52.428364+0000","last_peered":"2024-05-15T12:37:52.428364+0000","last_clean":"2024-05-15T12:37:52.428364+0000","last_became_active":"2024-05-15T12:37:13.095239+0000","last_became_peered":"2024-05-15T12:37:13.095239+0000","last_unstale":"2024-05-15T12:37:52.428364+0000","last_undegraded":"2024-05-15T12:37:52.428364+0000","last_fullsized":"2024-05-15T12:37:52.428364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:31:54.333216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433447+0000","last_change":"2024-05-15T12:37:15.108996+0000","last_active":"2024-05-15T12:37:52.433447+0000","last_peered":"2024-05-15T12:37:52.433447+0000","last_clean":"2024-05-15T12:37:52.433447+0000","last_became_active":"2024-05-15T12:37:15.108262+0000","last_became_peered":"2024-05-15T12:37:15.108262+0000","last_unstale":"2024-05-15T12:37:52.433447+0000","last_undegraded":"2024-05-15T12:37:52.433447+0000","last_fullsized":"2024-05-15T12:37:52.433447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:31:52.319671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428864+0000","last_change":"2024-05-15T12:37:13.096141+0000","last_active":"2024-05-15T12:37:52.428864+0000","last_peered":"2024-05-15T12:37:52.428864+0000","last_clean":"2024-05-15T12:37:52.428864+0000","last_became_active":"2024-05-15T12:37:13.095772+0000","last_became_peered":"2024-05-15T12:37:13.095772+0000","last_unstale":"2024-05-15T12:37:52.428864+0000","last_undegraded":"2024-05-15T12:37:52.428864+0000","last_fullsized":"2024-05-15T12:37:52.428864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:16:40.330004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434243+0000","last_change":"2024-05-15T12:37:09.069356+0000","last_active":"2024-05-15T12:37:52.434243+0000","last_peered":"2024-05-15T12:37:52.434243+0000","last_clean":"2024-05-15T12:37:52.434243+0000","last_became_active":"2024-05-15T12:37:09.069033+0000","last_became_peered":"2024-05-15T12:37:09.069033+0000","last_unstale":"2024-05-15T12:37:52.434243+0000","last_undegraded":"2024-05-15T12:37:52.434243+0000","last_fullsized":"2024-05-15T12:37:52.434243+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:35:33.517568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544605+0000","last_change":"2024-05-15T12:37:11.088397+0000","last_active":"2024-05-15T12:37:52.544605+0000","last_peered":"2024-05-15T12:37:52.544605+0000","last_clean":"2024-05-15T12:37:52.544605+0000","last_became_active":"2024-05-15T12:37:11.088067+0000","last_became_peered":"2024-05-15T12:37:11.088067+0000","last_unstale":"2024-05-15T12:37:52.544605+0000","last_undegraded":"2024-05-15T12:37:52.544605+0000","last_fullsized":"2024-05-15T12:37:52.544605+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:54:34.590074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319568+0000","last_change":"2024-05-15T12:37:15.107405+0000","last_active":"2024-05-15T12:37:52.319568+0000","last_peered":"2024-05-15T12:37:52.319568+0000","last_clean":"2024-05-15T12:37:52.319568+0000","last_became_active":"2024-05-15T12:37:15.107101+0000","last_became_peered":"2024-05-15T12:37:15.107101+0000","last_unstale":"2024-05-15T12:37:52.319568+0000","last_undegraded":"2024-05-15T12:37:52.319568+0000","last_fullsized":"2024-05-15T12:37:52.319568+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:04:19.513224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428908+0000","last_change":"2024-05-15T12:37:13.090322+0000","last_active":"2024-05-15T12:37:52.428908+0000","last_peered":"2024-05-15T12:37:52.428908+0000","last_clean":"2024-05-15T12:37:52.428908+0000","last_became_active":"2024-05-15T12:37:13.090014+0000","last_became_peered":"2024-05-15T12:37:13.090014+0000","last_unstale":"2024-05-15T12:37:52.428908+0000","last_undegraded":"2024-05-15T12:37:52.428908+0000","last_fullsized":"2024-05-15T12:37:52.428908+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:44:24.689515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420820+0000","last_change":"2024-05-15T12:37:09.068510+0000","last_active":"2024-05-15T12:37:52.420820+0000","last_peered":"2024-05-15T12:37:52.420820+0000","last_clean":"2024-05-15T12:37:52.420820+0000","last_became_active":"2024-05-15T12:37:09.068206+0000","last_became_peered":"2024-05-15T12:37:09.068206+0000","last_unstale":"2024-05-15T12:37:52.420820+0000","last_undegraded":"2024-05-15T12:37:52.420820+0000","last_fullsized":"2024-05-15T12:37:52.420820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:39:56.798950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320297+0000","last_change":"2024-05-15T12:37:11.084962+0000","last_active":"2024-05-15T12:37:52.320297+0000","last_peered":"2024-05-15T12:37:52.320297+0000","last_clean":"2024-05-15T12:37:52.320297+0000","last_became_active":"2024-05-15T12:37:11.084648+0000","last_became_peered":"2024-05-15T12:37:11.084648+0000","last_unstale":"2024-05-15T12:37:52.320297+0000","last_undegraded":"2024-05-15T12:37:52.320297+0000","last_fullsized":"2024-05-15T12:37:52.320297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:30:23.386442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545312+0000","last_change":"2024-05-15T12:37:15.107641+0000","last_active":"2024-05-15T12:37:52.545312+0000","last_peered":"2024-05-15T12:37:52.545312+0000","last_clean":"2024-05-15T12:37:52.545312+0000","last_became_active":"2024-05-15T12:37:15.107336+0000","last_became_peered":"2024-05-15T12:37:15.107336+0000","last_unstale":"2024-05-15T12:37:52.545312+0000","last_undegraded":"2024-05-15T12:37:52.545312+0000","last_fullsized":"2024-05-15T12:37:52.545312+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:55:52.656745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420602+0000","last_change":"2024-05-15T12:37:13.091923+0000","last_active":"2024-05-15T12:37:52.420602+0000","last_peered":"2024-05-15T12:37:52.420602+0000","last_clean":"2024-05-15T12:37:52.420602+0000","last_became_active":"2024-05-15T12:37:09.071579+0000","last_became_peered":"2024-05-15T12:37:09.071579+0000","last_unstale":"2024-05-15T12:37:52.420602+0000","last_undegraded":"2024-05-15T12:37:52.420602+0000","last_fullsized":"2024-05-15T12:37:52.420602+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:03:11.283684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049982009999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323592+0000","last_change":"2024-05-15T12:37:09.065661+0000","last_active":"2024-05-15T12:37:52.323592+0000","last_peered":"2024-05-15T12:37:52.323592+0000","last_clean":"2024-05-15T12:37:52.323592+0000","last_became_active":"2024-05-15T12:37:09.065283+0000","last_became_peered":"2024-05-15T12:37:09.065283+0000","last_unstale":"2024-05-15T12:37:52.323592+0000","last_undegraded":"2024-05-15T12:37:52.323592+0000","last_fullsized":"2024-05-15T12:37:52.323592+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:51:06.229965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'30","reported_seq":98,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129304+0000","last_change":"2024-05-15T12:37:11.083252+0000","last_active":"2024-05-15T12:38:29.129304+0000","last_peered":"2024-05-15T12:38:29.129304+0000","last_clean":"2024-05-15T12:38:29.129304+0000","last_became_active":"2024-05-15T12:37:11.082559+0000","last_became_peered":"2024-05-15T12:37:11.082559+0000","last_unstale":"2024-05-15T12:38:29.129304+0000","last_undegraded":"2024-05-15T12:38:29.129304+0000","last_fullsized":"2024-05-15T12:38:29.129304+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:27:09.894865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320646+0000","last_change":"2024-05-15T12:37:13.083190+0000","last_active":"2024-05-15T12:37:52.320646+0000","last_peered":"2024-05-15T12:37:52.320646+0000","last_clean":"2024-05-15T12:37:52.320646+0000","last_became_active":"2024-05-15T12:37:13.082778+0000","last_became_peered":"2024-05-15T12:37:13.082778+0000","last_unstale":"2024-05-15T12:37:52.320646+0000","last_undegraded":"2024-05-15T12:37:52.320646+0000","last_fullsized":"2024-05-15T12:37:52.320646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:43:52.008364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434354+0000","last_change":"2024-05-15T12:37:15.317087+0000","last_active":"2024-05-15T12:37:52.434354+0000","last_peered":"2024-05-15T12:37:52.434354+0000","last_clean":"2024-05-15T12:37:52.434354+0000","last_became_active":"2024-05-15T12:37:15.316784+0000","last_became_peered":"2024-05-15T12:37:15.316784+0000","last_unstale":"2024-05-15T12:37:52.434354+0000","last_undegraded":"2024-05-15T12:37:52.434354+0000","last_fullsized":"2024-05-15T12:37:52.434354+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:27:52.133545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323641+0000","last_change":"2024-05-15T12:37:09.070865+0000","last_active":"2024-05-15T12:37:52.323641+0000","last_peered":"2024-05-15T12:37:52.323641+0000","last_clean":"2024-05-15T12:37:52.323641+0000","last_became_active":"2024-05-15T12:37:09.070420+0000","last_became_peered":"2024-05-15T12:37:09.070420+0000","last_unstale":"2024-05-15T12:37:52.323641+0000","last_undegraded":"2024-05-15T12:37:52.323641+0000","last_fullsized":"2024-05-15T12:37:52.323641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:24:39.138873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428585+0000","last_change":"2024-05-15T12:37:13.086687+0000","last_active":"2024-05-15T12:37:52.428585+0000","last_peered":"2024-05-15T12:37:52.428585+0000","last_clean":"2024-05-15T12:37:52.428585+0000","last_became_active":"2024-05-15T12:37:09.063201+0000","last_became_peered":"2024-05-15T12:37:09.063201+0000","last_unstale":"2024-05-15T12:37:52.428585+0000","last_undegraded":"2024-05-15T12:37:52.428585+0000","last_fullsized":"2024-05-15T12:37:52.428585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:21:46.175922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0004594,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.325360+0000","last_change":"2024-05-15T12:37:11.079670+0000","last_active":"2024-05-15T12:37:52.325360+0000","last_peered":"2024-05-15T12:37:52.325360+0000","last_clean":"2024-05-15T12:37:52.325360+0000","last_became_active":"2024-05-15T12:37:11.078783+0000","last_became_peered":"2024-05-15T12:37:11.078783+0000","last_unstale":"2024-05-15T12:37:52.325360+0000","last_undegraded":"2024-05-15T12:37:52.325360+0000","last_fullsized":"2024-05-15T12:37:52.325360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:28:21.665496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428933+0000","last_change":"2024-05-15T12:37:13.091820+0000","last_active":"2024-05-15T12:37:52.428933+0000","last_peered":"2024-05-15T12:37:52.428933+0000","last_clean":"2024-05-15T12:37:52.428933+0000","last_became_active":"2024-05-15T12:37:13.091518+0000","last_became_peered":"2024-05-15T12:37:13.091518+0000","last_unstale":"2024-05-15T12:37:52.428933+0000","last_undegraded":"2024-05-15T12:37:52.428933+0000","last_fullsized":"2024-05-15T12:37:52.428933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:22:05.504481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421136+0000","last_change":"2024-05-15T12:37:15.104655+0000","last_active":"2024-05-15T12:37:52.421136+0000","last_peered":"2024-05-15T12:37:52.421136+0000","last_clean":"2024-05-15T12:37:52.421136+0000","last_became_active":"2024-05-15T12:37:15.104355+0000","last_became_peered":"2024-05-15T12:37:15.104355+0000","last_unstale":"2024-05-15T12:37:52.421136+0000","last_undegraded":"2024-05-15T12:37:52.421136+0000","last_fullsized":"2024-05-15T12:37:52.421136+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:24:33.872619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":345,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:54.416697+0000","last_change":"2024-05-15T12:36:59.870964+0000","last_active":"2024-05-15T12:37:54.416697+0000","last_peered":"2024-05-15T12:37:54.416697+0000","last_clean":"2024-05-15T12:37:54.416697+0000","last_became_active":"2024-05-15T12:36:59.870462+0000","last_became_peered":"2024-05-15T12:36:59.870462+0000","last_unstale":"2024-05-15T12:37:54.416697+0000","last_undegraded":"2024-05-15T12:37:54.416697+0000","last_fullsized":"2024-05-15T12:37:54.416697+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:45.857437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:45.857437+0000","last_clean_scrub_stamp":"2024-05-15T12:34:45.857437+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:41:02.759031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544666+0000","last_change":"2024-05-15T12:37:09.070440+0000","last_active":"2024-05-15T12:37:52.544666+0000","last_peered":"2024-05-15T12:37:52.544666+0000","last_clean":"2024-05-15T12:37:52.544666+0000","last_became_active":"2024-05-15T12:37:09.069855+0000","last_became_peered":"2024-05-15T12:37:09.069855+0000","last_unstale":"2024-05-15T12:37:52.544666+0000","last_undegraded":"2024-05-15T12:37:52.544666+0000","last_fullsized":"2024-05-15T12:37:52.544666+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:26:00.544976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129384+0000","last_change":"2024-05-15T12:37:11.084395+0000","last_active":"2024-05-15T12:38:29.129384+0000","last_peered":"2024-05-15T12:38:29.129384+0000","last_clean":"2024-05-15T12:38:29.129384+0000","last_became_active":"2024-05-15T12:37:11.084083+0000","last_became_peered":"2024-05-15T12:37:11.084083+0000","last_unstale":"2024-05-15T12:38:29.129384+0000","last_undegraded":"2024-05-15T12:38:29.129384+0000","last_fullsized":"2024-05-15T12:38:29.129384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:09:58.669195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433993+0000","last_change":"2024-05-15T12:37:13.100042+0000","last_active":"2024-05-15T12:37:52.433993+0000","last_peered":"2024-05-15T12:37:52.433993+0000","last_clean":"2024-05-15T12:37:52.433993+0000","last_became_active":"2024-05-15T12:37:13.099799+0000","last_became_peered":"2024-05-15T12:37:13.099799+0000","last_unstale":"2024-05-15T12:37:52.433993+0000","last_undegraded":"2024-05-15T12:37:52.433993+0000","last_fullsized":"2024-05-15T12:37:52.433993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:26:02.891879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324553+0000","last_change":"2024-05-15T12:37:15.103262+0000","last_active":"2024-05-15T12:37:52.324553+0000","last_peered":"2024-05-15T12:37:52.324553+0000","last_clean":"2024-05-15T12:37:52.324553+0000","last_became_active":"2024-05-15T12:37:15.102974+0000","last_became_peered":"2024-05-15T12:37:15.102974+0000","last_unstale":"2024-05-15T12:37:52.324553+0000","last_undegraded":"2024-05-15T12:37:52.324553+0000","last_fullsized":"2024-05-15T12:37:52.324553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:55:44.242107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319701+0000","last_change":"2024-05-15T12:37:09.066707+0000","last_active":"2024-05-15T12:37:52.319701+0000","last_peered":"2024-05-15T12:37:52.319701+0000","last_clean":"2024-05-15T12:37:52.319701+0000","last_became_active":"2024-05-15T12:37:09.066314+0000","last_became_peered":"2024-05-15T12:37:09.066314+0000","last_unstale":"2024-05-15T12:37:52.319701+0000","last_undegraded":"2024-05-15T12:37:52.319701+0000","last_fullsized":"2024-05-15T12:37:52.319701+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:00:57.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":125,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:36.885247+0000","last_change":"2024-05-15T12:37:13.088457+0000","last_active":"2024-05-15T12:38:36.885247+0000","last_peered":"2024-05-15T12:38:36.885247+0000","last_clean":"2024-05-15T12:38:36.885247+0000","last_became_active":"2024-05-15T12:37:09.074657+0000","last_became_peered":"2024-05-15T12:37:09.074657+0000","last_unstale":"2024-05-15T12:38:36.885247+0000","last_undegraded":"2024-05-15T12:38:36.885247+0000","last_fullsized":"2024-05-15T12:38:36.885247+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:05:56.605060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080141299999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320770+0000","last_change":"2024-05-15T12:37:11.077982+0000","last_active":"2024-05-15T12:37:52.320770+0000","last_peered":"2024-05-15T12:37:52.320770+0000","last_clean":"2024-05-15T12:37:52.320770+0000","last_became_active":"2024-05-15T12:37:11.077659+0000","last_became_peered":"2024-05-15T12:37:11.077659+0000","last_unstale":"2024-05-15T12:37:52.320770+0000","last_undegraded":"2024-05-15T12:37:52.320770+0000","last_fullsized":"2024-05-15T12:37:52.320770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:25:39.482645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421258+0000","last_change":"2024-05-15T12:37:13.096395+0000","last_active":"2024-05-15T12:37:52.421258+0000","last_peered":"2024-05-15T12:37:52.421258+0000","last_clean":"2024-05-15T12:37:52.421258+0000","last_became_active":"2024-05-15T12:37:13.096105+0000","last_became_peered":"2024-05-15T12:37:13.096105+0000","last_unstale":"2024-05-15T12:37:52.421258+0000","last_undegraded":"2024-05-15T12:37:52.421258+0000","last_fullsized":"2024-05-15T12:37:52.421258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:18:11.570497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323475+0000","last_change":"2024-05-15T12:37:15.319169+0000","last_active":"2024-05-15T12:37:52.323475+0000","last_peered":"2024-05-15T12:37:52.323475+0000","last_clean":"2024-05-15T12:37:52.323475+0000","last_became_active":"2024-05-15T12:37:15.318837+0000","last_became_peered":"2024-05-15T12:37:15.318837+0000","last_unstale":"2024-05-15T12:37:52.323475+0000","last_undegraded":"2024-05-15T12:37:52.323475+0000","last_fullsized":"2024-05-15T12:37:52.323475+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:53:26.379829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324783+0000","last_change":"2024-05-15T12:37:09.072210+0000","last_active":"2024-05-15T12:37:52.324783+0000","last_peered":"2024-05-15T12:37:52.324783+0000","last_clean":"2024-05-15T12:37:52.324783+0000","last_became_active":"2024-05-15T12:37:09.071893+0000","last_became_peered":"2024-05-15T12:37:09.071893+0000","last_unstale":"2024-05-15T12:37:52.324783+0000","last_undegraded":"2024-05-15T12:37:52.324783+0000","last_fullsized":"2024-05-15T12:37:52.324783+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:35:55.640503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320225+0000","last_change":"2024-05-15T12:37:11.078505+0000","last_active":"2024-05-15T12:37:52.320225+0000","last_peered":"2024-05-15T12:37:52.320225+0000","last_clean":"2024-05-15T12:37:52.320225+0000","last_became_active":"2024-05-15T12:37:11.078048+0000","last_became_peered":"2024-05-15T12:37:11.078048+0000","last_unstale":"2024-05-15T12:37:52.320225+0000","last_undegraded":"2024-05-15T12:37:52.320225+0000","last_fullsized":"2024-05-15T12:37:52.320225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:12:02.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544606+0000","last_change":"2024-05-15T12:37:13.097882+0000","last_active":"2024-05-15T12:37:52.544606+0000","last_peered":"2024-05-15T12:37:52.544606+0000","last_clean":"2024-05-15T12:37:52.544606+0000","last_became_active":"2024-05-15T12:37:13.097612+0000","last_became_peered":"2024-05-15T12:37:13.097612+0000","last_unstale":"2024-05-15T12:37:52.544606+0000","last_undegraded":"2024-05-15T12:37:52.544606+0000","last_fullsized":"2024-05-15T12:37:52.544606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:34:37.770323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319498+0000","last_change":"2024-05-15T12:37:15.100983+0000","last_active":"2024-05-15T12:37:52.319498+0000","last_peered":"2024-05-15T12:37:52.319498+0000","last_clean":"2024-05-15T12:37:52.319498+0000","last_became_active":"2024-05-15T12:37:15.100662+0000","last_became_peered":"2024-05-15T12:37:15.100662+0000","last_unstale":"2024-05-15T12:37:52.319498+0000","last_undegraded":"2024-05-15T12:37:52.319498+0000","last_fullsized":"2024-05-15T12:37:52.319498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:14:28.954709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421142+0000","last_change":"2024-05-15T12:37:09.067662+0000","last_active":"2024-05-15T12:37:52.421142+0000","last_peered":"2024-05-15T12:37:52.421142+0000","last_clean":"2024-05-15T12:37:52.421142+0000","last_became_active":"2024-05-15T12:37:09.067101+0000","last_became_peered":"2024-05-15T12:37:09.067101+0000","last_unstale":"2024-05-15T12:37:52.421142+0000","last_undegraded":"2024-05-15T12:37:52.421142+0000","last_fullsized":"2024-05-15T12:37:52.421142+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:21:22.349984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428547+0000","last_change":"2024-05-15T12:37:24.674107+0000","last_active":"2024-05-15T12:37:52.428547+0000","last_peered":"2024-05-15T12:37:52.428547+0000","last_clean":"2024-05-15T12:37:52.428547+0000","last_became_active":"2024-05-15T12:37:24.666449+0000","last_became_peered":"2024-05-15T12:37:24.666449+0000","last_unstale":"2024-05-15T12:37:52.428547+0000","last_undegraded":"2024-05-15T12:37:52.428547+0000","last_fullsized":"2024-05-15T12:37:52.428547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:52:59.925765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"68'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129254+0000","last_change":"2024-05-15T12:37:13.096029+0000","last_active":"2024-05-15T12:38:29.129254+0000","last_peered":"2024-05-15T12:38:29.129254+0000","last_clean":"2024-05-15T12:38:29.129254+0000","last_became_active":"2024-05-15T12:37:13.095733+0000","last_became_peered":"2024-05-15T12:37:13.095733+0000","last_unstale":"2024-05-15T12:38:29.129254+0000","last_undegraded":"2024-05-15T12:38:29.129254+0000","last_fullsized":"2024-05-15T12:38:29.129254+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:36:00.171088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433984+0000","last_change":"2024-05-15T12:37:15.317846+0000","last_active":"2024-05-15T12:37:52.433984+0000","last_peered":"2024-05-15T12:37:52.433984+0000","last_clean":"2024-05-15T12:37:52.433984+0000","last_became_active":"2024-05-15T12:37:15.317545+0000","last_became_peered":"2024-05-15T12:37:15.317545+0000","last_unstale":"2024-05-15T12:37:52.433984+0000","last_undegraded":"2024-05-15T12:37:52.433984+0000","last_fullsized":"2024-05-15T12:37:52.433984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:36:07.246378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320791+0000","last_change":"2024-05-15T12:37:09.072381+0000","last_active":"2024-05-15T12:37:52.320791+0000","last_peered":"2024-05-15T12:37:52.320791+0000","last_clean":"2024-05-15T12:37:52.320791+0000","last_became_active":"2024-05-15T12:37:09.072107+0000","last_became_peered":"2024-05-15T12:37:09.072107+0000","last_unstale":"2024-05-15T12:37:52.320791+0000","last_undegraded":"2024-05-15T12:37:52.320791+0000","last_fullsized":"2024-05-15T12:37:52.320791+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:08:17.775714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323191+0000","last_change":"2024-05-15T12:37:11.085680+0000","last_active":"2024-05-15T12:37:52.323191+0000","last_peered":"2024-05-15T12:37:52.323191+0000","last_clean":"2024-05-15T12:37:52.323191+0000","last_became_active":"2024-05-15T12:37:11.085365+0000","last_became_peered":"2024-05-15T12:37:11.085365+0000","last_unstale":"2024-05-15T12:37:52.323191+0000","last_undegraded":"2024-05-15T12:37:52.323191+0000","last_fullsized":"2024-05-15T12:37:52.323191+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:22:04.818789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129290+0000","last_change":"2024-05-15T12:37:13.091600+0000","last_active":"2024-05-15T12:38:29.129290+0000","last_peered":"2024-05-15T12:38:29.129290+0000","last_clean":"2024-05-15T12:38:29.129290+0000","last_became_active":"2024-05-15T12:37:13.091320+0000","last_became_peered":"2024-05-15T12:37:13.091320+0000","last_unstale":"2024-05-15T12:38:29.129290+0000","last_undegraded":"2024-05-15T12:38:29.129290+0000","last_fullsized":"2024-05-15T12:38:29.129290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:14:01.722227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323196+0000","last_change":"2024-05-15T12:37:15.103175+0000","last_active":"2024-05-15T12:37:52.323196+0000","last_peered":"2024-05-15T12:37:52.323196+0000","last_clean":"2024-05-15T12:37:52.323196+0000","last_became_active":"2024-05-15T12:37:15.102888+0000","last_became_peered":"2024-05-15T12:37:15.102888+0000","last_unstale":"2024-05-15T12:37:52.323196+0000","last_undegraded":"2024-05-15T12:37:52.323196+0000","last_fullsized":"2024-05-15T12:37:52.323196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:06:55.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545675+0000","last_change":"2024-05-15T12:37:09.073873+0000","last_active":"2024-05-15T12:37:52.545675+0000","last_peered":"2024-05-15T12:37:52.545675+0000","last_clean":"2024-05-15T12:37:52.545675+0000","last_became_active":"2024-05-15T12:37:09.073514+0000","last_became_peered":"2024-05-15T12:37:09.073514+0000","last_unstale":"2024-05-15T12:37:52.545675+0000","last_undegraded":"2024-05-15T12:37:52.545675+0000","last_fullsized":"2024-05-15T12:37:52.545675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:29:09.850307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544326+0000","last_change":"2024-05-15T12:37:11.081168+0000","last_active":"2024-05-15T12:37:52.544326+0000","last_peered":"2024-05-15T12:37:52.544326+0000","last_clean":"2024-05-15T12:37:52.544326+0000","last_became_active":"2024-05-15T12:37:11.080867+0000","last_became_peered":"2024-05-15T12:37:11.080867+0000","last_unstale":"2024-05-15T12:37:52.544326+0000","last_undegraded":"2024-05-15T12:37:52.544326+0000","last_fullsized":"2024-05-15T12:37:52.544326+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:57:58.948283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.322971+0000","last_change":"2024-05-15T12:37:13.091686+0000","last_active":"2024-05-15T12:37:52.322971+0000","last_peered":"2024-05-15T12:37:52.322971+0000","last_clean":"2024-05-15T12:37:52.322971+0000","last_became_active":"2024-05-15T12:37:13.091091+0000","last_became_peered":"2024-05-15T12:37:13.091091+0000","last_unstale":"2024-05-15T12:37:52.322971+0000","last_undegraded":"2024-05-15T12:37:52.322971+0000","last_fullsized":"2024-05-15T12:37:52.322971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:11:24.693744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420896+0000","last_change":"2024-05-15T12:37:15.109231+0000","last_active":"2024-05-15T12:37:52.420896+0000","last_peered":"2024-05-15T12:37:52.420896+0000","last_clean":"2024-05-15T12:37:52.420896+0000","last_became_active":"2024-05-15T12:37:15.108923+0000","last_became_peered":"2024-05-15T12:37:15.108923+0000","last_unstale":"2024-05-15T12:37:52.420896+0000","last_undegraded":"2024-05-15T12:37:52.420896+0000","last_fullsized":"2024-05-15T12:37:52.420896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:28:25.954595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129386+0000","last_change":"2024-05-15T12:37:13.091283+0000","last_active":"2024-05-15T12:38:29.129386+0000","last_peered":"2024-05-15T12:38:29.129386+0000","last_clean":"2024-05-15T12:38:29.129386+0000","last_became_active":"2024-05-15T12:37:13.091000+0000","last_became_peered":"2024-05-15T12:37:13.091000+0000","last_unstale":"2024-05-15T12:38:29.129386+0000","last_undegraded":"2024-05-15T12:38:29.129386+0000","last_fullsized":"2024-05-15T12:38:29.129386+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:13:24.211092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544263+0000","last_change":"2024-05-15T12:37:09.069252+0000","last_active":"2024-05-15T12:37:52.544263+0000","last_peered":"2024-05-15T12:37:52.544263+0000","last_clean":"2024-05-15T12:37:52.544263+0000","last_became_active":"2024-05-15T12:37:09.068907+0000","last_became_peered":"2024-05-15T12:37:09.068907+0000","last_unstale":"2024-05-15T12:37:52.544263+0000","last_undegraded":"2024-05-15T12:37:52.544263+0000","last_fullsized":"2024-05-15T12:37:52.544263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:55:58.609247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.325419+0000","last_change":"2024-05-15T12:37:11.084675+0000","last_active":"2024-05-15T12:37:52.325419+0000","last_peered":"2024-05-15T12:37:52.325419+0000","last_clean":"2024-05-15T12:37:52.325419+0000","last_became_active":"2024-05-15T12:37:11.084429+0000","last_became_peered":"2024-05-15T12:37:11.084429+0000","last_unstale":"2024-05-15T12:37:52.325419+0000","last_undegraded":"2024-05-15T12:37:52.325419+0000","last_fullsized":"2024-05-15T12:37:52.325419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:39:38.526026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544974+0000","last_change":"2024-05-15T12:37:15.316778+0000","last_active":"2024-05-15T12:37:52.544974+0000","last_peered":"2024-05-15T12:37:52.544974+0000","last_clean":"2024-05-15T12:37:52.544974+0000","last_became_active":"2024-05-15T12:37:15.316351+0000","last_became_peered":"2024-05-15T12:37:15.316351+0000","last_unstale":"2024-05-15T12:37:52.544974+0000","last_undegraded":"2024-05-15T12:37:52.544974+0000","last_fullsized":"2024-05-15T12:37:52.544974+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:05:09.440865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421244+0000","last_change":"2024-05-15T12:37:13.094722+0000","last_active":"2024-05-15T12:37:52.421244+0000","last_peered":"2024-05-15T12:37:52.421244+0000","last_clean":"2024-05-15T12:37:52.421244+0000","last_became_active":"2024-05-15T12:37:13.094410+0000","last_became_peered":"2024-05-15T12:37:13.094410+0000","last_unstale":"2024-05-15T12:37:52.421244+0000","last_undegraded":"2024-05-15T12:37:52.421244+0000","last_fullsized":"2024-05-15T12:37:52.421244+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:39:29.669553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323076+0000","last_change":"2024-05-15T12:37:09.071534+0000","last_active":"2024-05-15T12:37:52.323076+0000","last_peered":"2024-05-15T12:37:52.323076+0000","last_clean":"2024-05-15T12:37:52.323076+0000","last_became_active":"2024-05-15T12:37:09.071219+0000","last_became_peered":"2024-05-15T12:37:09.071219+0000","last_unstale":"2024-05-15T12:37:52.323076+0000","last_undegraded":"2024-05-15T12:37:52.323076+0000","last_fullsized":"2024-05-15T12:37:52.323076+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:00:16.664754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324097+0000","last_change":"2024-05-15T12:37:11.087249+0000","last_active":"2024-05-15T12:37:52.324097+0000","last_peered":"2024-05-15T12:37:52.324097+0000","last_clean":"2024-05-15T12:37:52.324097+0000","last_became_active":"2024-05-15T12:37:11.086645+0000","last_became_peered":"2024-05-15T12:37:11.086645+0000","last_unstale":"2024-05-15T12:37:52.324097+0000","last_undegraded":"2024-05-15T12:37:52.324097+0000","last_fullsized":"2024-05-15T12:37:52.324097+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:19:27.815519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.429361+0000","last_change":"2024-05-15T12:37:15.103029+0000","last_active":"2024-05-15T12:37:52.429361+0000","last_peered":"2024-05-15T12:37:52.429361+0000","last_clean":"2024-05-15T12:37:52.429361+0000","last_became_active":"2024-05-15T12:37:15.102711+0000","last_became_peered":"2024-05-15T12:37:15.102711+0000","last_unstale":"2024-05-15T12:37:52.429361+0000","last_undegraded":"2024-05-15T12:37:52.429361+0000","last_fullsized":"2024-05-15T12:37:52.429361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:03:40.305588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.325084+0000","last_change":"2024-05-15T12:37:13.083388+0000","last_active":"2024-05-15T12:37:52.325084+0000","last_peered":"2024-05-15T12:37:52.325084+0000","last_clean":"2024-05-15T12:37:52.325084+0000","last_became_active":"2024-05-15T12:37:13.082933+0000","last_became_peered":"2024-05-15T12:37:13.082933+0000","last_unstale":"2024-05-15T12:37:52.325084+0000","last_undegraded":"2024-05-15T12:37:52.325084+0000","last_fullsized":"2024-05-15T12:37:52.325084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:15:53.954674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544350+0000","last_change":"2024-05-15T12:37:09.067077+0000","last_active":"2024-05-15T12:37:52.544350+0000","last_peered":"2024-05-15T12:37:52.544350+0000","last_clean":"2024-05-15T12:37:52.544350+0000","last_became_active":"2024-05-15T12:37:09.066797+0000","last_became_peered":"2024-05-15T12:37:09.066797+0000","last_unstale":"2024-05-15T12:37:52.544350+0000","last_undegraded":"2024-05-15T12:37:52.544350+0000","last_fullsized":"2024-05-15T12:37:52.544350+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:25:58.827990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323595+0000","last_change":"2024-05-15T12:37:11.087288+0000","last_active":"2024-05-15T12:37:52.323595+0000","last_peered":"2024-05-15T12:37:52.323595+0000","last_clean":"2024-05-15T12:37:52.323595+0000","last_became_active":"2024-05-15T12:37:11.086950+0000","last_became_peered":"2024-05-15T12:37:11.086950+0000","last_unstale":"2024-05-15T12:37:52.323595+0000","last_undegraded":"2024-05-15T12:37:52.323595+0000","last_fullsized":"2024-05-15T12:37:52.323595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:56:00.337217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324257+0000","last_change":"2024-05-15T12:37:15.102502+0000","last_active":"2024-05-15T12:37:52.324257+0000","last_peered":"2024-05-15T12:37:52.324257+0000","last_clean":"2024-05-15T12:37:52.324257+0000","last_became_active":"2024-05-15T12:37:15.102133+0000","last_became_peered":"2024-05-15T12:37:15.102133+0000","last_unstale":"2024-05-15T12:37:52.324257+0000","last_undegraded":"2024-05-15T12:37:52.324257+0000","last_fullsized":"2024-05-15T12:37:52.324257+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:09:39.802079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:29.129276+0000","last_change":"2024-05-15T12:37:13.093833+0000","last_active":"2024-05-15T12:38:29.129276+0000","last_peered":"2024-05-15T12:38:29.129276+0000","last_clean":"2024-05-15T12:38:29.129276+0000","last_became_active":"2024-05-15T12:37:13.093530+0000","last_became_peered":"2024-05-15T12:37:13.093530+0000","last_unstale":"2024-05-15T12:38:29.129276+0000","last_undegraded":"2024-05-15T12:38:29.129276+0000","last_fullsized":"2024-05-15T12:38:29.129276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:39:41.201865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"60'1","reported_seq":41,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544770+0000","last_change":"2024-05-15T12:37:09.067314+0000","last_active":"2024-05-15T12:37:52.544770+0000","last_peered":"2024-05-15T12:37:52.544770+0000","last_clean":"2024-05-15T12:37:52.544770+0000","last_became_active":"2024-05-15T12:37:09.066974+0000","last_became_peered":"2024-05-15T12:37:09.066974+0000","last_unstale":"2024-05-15T12:37:52.544770+0000","last_undegraded":"2024-05-15T12:37:52.544770+0000","last_fullsized":"2024-05-15T12:37:52.544770+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:47:05.111867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323640+0000","last_change":"2024-05-15T12:37:11.083458+0000","last_active":"2024-05-15T12:37:52.323640+0000","last_peered":"2024-05-15T12:37:52.323640+0000","last_clean":"2024-05-15T12:37:52.323640+0000","last_became_active":"2024-05-15T12:37:11.083080+0000","last_became_peered":"2024-05-15T12:37:11.083080+0000","last_unstale":"2024-05-15T12:37:52.323640+0000","last_undegraded":"2024-05-15T12:37:52.323640+0000","last_fullsized":"2024-05-15T12:37:52.323640+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:08:45.744306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712680,"statfs":{"total":95990841344,"available":95961784320,"internally_reserved":0,"allocated":1392640,"data_stored":923479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708536,"statfs":{"total":95990841344,"available":95957540864,"internally_reserved":0,"allocated":1372160,"data_stored":922427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713224,"statfs":{"total":95990841344,"available":95962341376,"internally_reserved":0,"allocated":831488,"data_stored":333226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28380,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712676,"statfs":{"total":95990841344,"available":95961780224,"internally_reserved":0,"allocated":1396736,"data_stored":923698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713280,"statfs":{"total":95990841344,"available":95962398720,"internally_reserved":0,"allocated":774144,"data_stored":331886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713236,"statfs":{"total":95990841344,"available":95962353664,"internally_reserved":0,"allocated":819200,"data_stored":333440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018918,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713240,"statfs":{"total":95990841344,"available":95962357760,"internally_reserved":0,"allocated":815104,"data_stored":333552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713268,"statfs":{"total":95990841344,"available":95962386432,"internally_reserved":0,"allocated":786432,"data_stored":332110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:38:42.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-15T12:38:42.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:42 smithi173 ceph-mon[26787]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:38:42.902 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:42 smithi102 bash[60793]: Getting image source signatures 2024-05-15T12:38:42.902 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:42 smithi102 bash[60793]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-05-15T12:38:42.902 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:42 smithi102 bash[60793]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-05-15T12:38:42.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:42 smithi102 bash[60793]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-05-15T12:38:42.945 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:43.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:43 smithi173 ceph-mon[26787]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:43.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:43 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:43.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:38:43 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:38:43.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:43 smithi102 ceph-mon[31347]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:43.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:43 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:43.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:43 smithi102 ceph-mon[24799]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:43.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:43 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:43.653 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:43 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:38:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:38:44.555 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:44 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:44.555 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:44 smithi102 bash[60793]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-05-15T12:38:44.555 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:44 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:44.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:44 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:44.857 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:44 smithi102 bash[60793]: Writing manifest to image destination 2024-05-15T12:38:45.118 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:44 smithi102 podman[60793]: 2024-05-15 12:38:44.894072194 +0000 UTC m=+3.037771875 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-05-15T12:38:45.118 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 podman[60793]: 2024-05-15 12:38:45.009065389 +0000 UTC m=+3.152765078 container create 3ec530770871f0cfeaf21b78ee51a2f12e8904e2d2bcfc5514645deb4ceadfd6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-15T12:38:45.502 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-mon[24799]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:45.503 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:45 smithi102 ceph-mon[31347]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:45.733 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:38:45.735 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-05-15T12:38:45.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:45 smithi173 ceph-mon[26787]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:45.902 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 podman[60793]: 2024-05-15 12:38:45.656982514 +0000 UTC m=+3.800682202 container init 3ec530770871f0cfeaf21b78ee51a2f12e8904e2d2bcfc5514645deb4ceadfd6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 podman[60793]: 2024-05-15 12:38:45.660485146 +0000 UTC m=+3.804184816 container start 3ec530770871f0cfeaf21b78ee51a2f12e8904e2d2bcfc5514645deb4ceadfd6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.664Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.664Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.664Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.664Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=arp 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=bcache 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=bonding 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=btrfs 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=conntrack 2024-05-15T12:38:45.903 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=cpu 2024-05-15T12:38:45.904 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-05-15T12:38:45.904 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=diskstats 2024-05-15T12:38:45.904 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=dmi 2024-05-15T12:38:45.904 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=edac 2024-05-15T12:38:45.904 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=entropy 2024-05-15T12:38:45.904 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-05-15T12:38:45.904 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=filefd 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=filesystem 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=hwmon 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=infiniband 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=ipvs 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=loadavg 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=mdadm 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=meminfo 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=netclass 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=netdev 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=netstat 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=nfs 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=nfsd 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=nvme 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=os 2024-05-15T12:38:45.905 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=pressure 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=rapl 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=schedstat 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=selinux 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=sockstat 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=softnet 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=stat 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=tapestats 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=textfile 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=time 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=uname 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=vmstat 2024-05-15T12:38:45.906 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=xfs 2024-05-15T12:38:45.907 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.665Z caller=node_exporter.go:117 level=info collector=zfs 2024-05-15T12:38:45.907 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.666Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-05-15T12:38:45.907 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a[60971]: ts=2024-05-15T12:38:45.666Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-05-15T12:38:45.907 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 bash[60793]: 3ec530770871f0cfeaf21b78ee51a2f12e8904e2d2bcfc5514645deb4ceadfd6 2024-05-15T12:38:45.907 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:38:45 smithi102 systemd[1]: Started Ceph node-exporter.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:38:46.556 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-05-15T12:38:44.341682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1067,"num_read_kb":910,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":686,"ondisk_log_size":686,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":228308,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":749700140,"statfs":{"total":767926730752,"available":767692943360,"internally_reserved":0,"allocated":8187904,"data_stored":4433818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":225496625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002411"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323082+0000","last_change":"2024-05-15T12:37:15.102600+0000","last_active":"2024-05-15T12:37:52.323082+0000","last_peered":"2024-05-15T12:37:52.323082+0000","last_clean":"2024-05-15T12:37:52.323082+0000","last_became_active":"2024-05-15T12:37:15.102241+0000","last_became_peered":"2024-05-15T12:37:15.102241+0000","last_unstale":"2024-05-15T12:37:52.323082+0000","last_undegraded":"2024-05-15T12:37:52.323082+0000","last_fullsized":"2024-05-15T12:37:52.323082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:06:20.143991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323627+0000","last_change":"2024-05-15T12:37:13.092914+0000","last_active":"2024-05-15T12:37:52.323627+0000","last_peered":"2024-05-15T12:37:52.323627+0000","last_clean":"2024-05-15T12:37:52.323627+0000","last_became_active":"2024-05-15T12:37:13.092398+0000","last_became_peered":"2024-05-15T12:37:13.092398+0000","last_unstale":"2024-05-15T12:37:52.323627+0000","last_undegraded":"2024-05-15T12:37:52.323627+0000","last_fullsized":"2024-05-15T12:37:52.323627+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:17:08.171640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320990+0000","last_change":"2024-05-15T12:37:09.071444+0000","last_active":"2024-05-15T12:37:52.320990+0000","last_peered":"2024-05-15T12:37:52.320990+0000","last_clean":"2024-05-15T12:37:52.320990+0000","last_became_active":"2024-05-15T12:37:09.071161+0000","last_became_peered":"2024-05-15T12:37:09.071161+0000","last_unstale":"2024-05-15T12:37:52.320990+0000","last_undegraded":"2024-05-15T12:37:52.320990+0000","last_fullsized":"2024-05-15T12:37:52.320990+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:35:51.483393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.322873+0000","last_change":"2024-05-15T12:37:24.678381+0000","last_active":"2024-05-15T12:37:52.322873+0000","last_peered":"2024-05-15T12:37:52.322873+0000","last_clean":"2024-05-15T12:37:52.322873+0000","last_became_active":"2024-05-15T12:37:24.667945+0000","last_became_peered":"2024-05-15T12:37:24.667945+0000","last_unstale":"2024-05-15T12:37:52.322873+0000","last_undegraded":"2024-05-15T12:37:52.322873+0000","last_fullsized":"2024-05-15T12:37:52.322873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:33:49.381866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544670+0000","last_change":"2024-05-15T12:37:15.319547+0000","last_active":"2024-05-15T12:37:52.544670+0000","last_peered":"2024-05-15T12:37:52.544670+0000","last_clean":"2024-05-15T12:37:52.544670+0000","last_became_active":"2024-05-15T12:37:15.319041+0000","last_became_peered":"2024-05-15T12:37:15.319041+0000","last_unstale":"2024-05-15T12:37:52.544670+0000","last_undegraded":"2024-05-15T12:37:52.544670+0000","last_fullsized":"2024-05-15T12:37:52.544670+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:14:54.874371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.322907+0000","last_change":"2024-05-15T12:37:13.093049+0000","last_active":"2024-05-15T12:37:52.322907+0000","last_peered":"2024-05-15T12:37:52.322907+0000","last_clean":"2024-05-15T12:37:52.322907+0000","last_became_active":"2024-05-15T12:37:13.092704+0000","last_became_peered":"2024-05-15T12:37:13.092704+0000","last_unstale":"2024-05-15T12:37:52.322907+0000","last_undegraded":"2024-05-15T12:37:52.322907+0000","last_fullsized":"2024-05-15T12:37:52.322907+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:03:40.608770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545470+0000","last_change":"2024-05-15T12:37:09.065293+0000","last_active":"2024-05-15T12:37:52.545470+0000","last_peered":"2024-05-15T12:37:52.545470+0000","last_clean":"2024-05-15T12:37:52.545470+0000","last_became_active":"2024-05-15T12:37:09.065011+0000","last_became_peered":"2024-05-15T12:37:09.065011+0000","last_unstale":"2024-05-15T12:37:52.545470+0000","last_undegraded":"2024-05-15T12:37:52.545470+0000","last_fullsized":"2024-05-15T12:37:52.545470+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:37:59.896309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544414+0000","last_change":"2024-05-15T12:37:11.084240+0000","last_active":"2024-05-15T12:37:52.544414+0000","last_peered":"2024-05-15T12:37:52.544414+0000","last_clean":"2024-05-15T12:37:52.544414+0000","last_became_active":"2024-05-15T12:37:11.083917+0000","last_became_peered":"2024-05-15T12:37:11.083917+0000","last_unstale":"2024-05-15T12:37:52.544414+0000","last_undegraded":"2024-05-15T12:37:52.544414+0000","last_fullsized":"2024-05-15T12:37:52.544414+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:21:16.143398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319786+0000","last_change":"2024-05-15T12:37:15.106034+0000","last_active":"2024-05-15T12:37:52.319786+0000","last_peered":"2024-05-15T12:37:52.319786+0000","last_clean":"2024-05-15T12:37:52.319786+0000","last_became_active":"2024-05-15T12:37:15.105669+0000","last_became_peered":"2024-05-15T12:37:15.105669+0000","last_unstale":"2024-05-15T12:37:52.319786+0000","last_undegraded":"2024-05-15T12:37:52.319786+0000","last_fullsized":"2024-05-15T12:37:52.319786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:36:23.541420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420635+0000","last_change":"2024-05-15T12:37:13.098122+0000","last_active":"2024-05-15T12:37:52.420635+0000","last_peered":"2024-05-15T12:37:52.420635+0000","last_clean":"2024-05-15T12:37:52.420635+0000","last_became_active":"2024-05-15T12:37:13.097726+0000","last_became_peered":"2024-05-15T12:37:13.097726+0000","last_unstale":"2024-05-15T12:37:52.420635+0000","last_undegraded":"2024-05-15T12:37:52.420635+0000","last_fullsized":"2024-05-15T12:37:52.420635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:23:01.705026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421017+0000","last_change":"2024-05-15T12:37:09.067399+0000","last_active":"2024-05-15T12:37:52.421017+0000","last_peered":"2024-05-15T12:37:52.421017+0000","last_clean":"2024-05-15T12:37:52.421017+0000","last_became_active":"2024-05-15T12:37:09.066761+0000","last_became_peered":"2024-05-15T12:37:09.066761+0000","last_unstale":"2024-05-15T12:37:52.421017+0000","last_undegraded":"2024-05-15T12:37:52.421017+0000","last_fullsized":"2024-05-15T12:37:52.421017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:07:16.154974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323252+0000","last_change":"2024-05-15T12:37:11.084043+0000","last_active":"2024-05-15T12:37:52.323252+0000","last_peered":"2024-05-15T12:37:52.323252+0000","last_clean":"2024-05-15T12:37:52.323252+0000","last_became_active":"2024-05-15T12:37:11.083710+0000","last_became_peered":"2024-05-15T12:37:11.083710+0000","last_unstale":"2024-05-15T12:37:52.323252+0000","last_undegraded":"2024-05-15T12:37:52.323252+0000","last_fullsized":"2024-05-15T12:37:52.323252+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:02:47.284700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420524+0000","last_change":"2024-05-15T12:37:15.103486+0000","last_active":"2024-05-15T12:37:52.420524+0000","last_peered":"2024-05-15T12:37:52.420524+0000","last_clean":"2024-05-15T12:37:52.420524+0000","last_became_active":"2024-05-15T12:37:15.103189+0000","last_became_peered":"2024-05-15T12:37:15.103189+0000","last_unstale":"2024-05-15T12:37:52.420524+0000","last_undegraded":"2024-05-15T12:37:52.420524+0000","last_fullsized":"2024-05-15T12:37:52.420524+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:59:24.916554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433597+0000","last_change":"2024-05-15T12:37:13.094897+0000","last_active":"2024-05-15T12:37:52.433597+0000","last_peered":"2024-05-15T12:37:52.433597+0000","last_clean":"2024-05-15T12:37:52.433597+0000","last_became_active":"2024-05-15T12:37:13.094582+0000","last_became_peered":"2024-05-15T12:37:13.094582+0000","last_unstale":"2024-05-15T12:37:52.433597+0000","last_undegraded":"2024-05-15T12:37:52.433597+0000","last_fullsized":"2024-05-15T12:37:52.433597+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:54:48.816395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420966+0000","last_change":"2024-05-15T12:37:09.070511+0000","last_active":"2024-05-15T12:37:52.420966+0000","last_peered":"2024-05-15T12:37:52.420966+0000","last_clean":"2024-05-15T12:37:52.420966+0000","last_became_active":"2024-05-15T12:37:09.070210+0000","last_became_peered":"2024-05-15T12:37:09.070210+0000","last_unstale":"2024-05-15T12:37:52.420966+0000","last_undegraded":"2024-05-15T12:37:52.420966+0000","last_fullsized":"2024-05-15T12:37:52.420966+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:08:47.204599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323749+0000","last_change":"2024-05-15T12:37:11.087129+0000","last_active":"2024-05-15T12:37:52.323749+0000","last_peered":"2024-05-15T12:37:52.323749+0000","last_clean":"2024-05-15T12:37:52.323749+0000","last_became_active":"2024-05-15T12:37:11.086490+0000","last_became_peered":"2024-05-15T12:37:11.086490+0000","last_unstale":"2024-05-15T12:37:52.323749+0000","last_undegraded":"2024-05-15T12:37:52.323749+0000","last_fullsized":"2024-05-15T12:37:52.323749+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:28:13.748140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324126+0000","last_change":"2024-05-15T12:37:15.316887+0000","last_active":"2024-05-15T12:37:52.324126+0000","last_peered":"2024-05-15T12:37:52.324126+0000","last_clean":"2024-05-15T12:37:52.324126+0000","last_became_active":"2024-05-15T12:37:15.316504+0000","last_became_peered":"2024-05-15T12:37:15.316504+0000","last_unstale":"2024-05-15T12:37:52.324126+0000","last_undegraded":"2024-05-15T12:37:52.324126+0000","last_fullsized":"2024-05-15T12:37:52.324126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:49:58.474167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324615+0000","last_change":"2024-05-15T12:37:13.083240+0000","last_active":"2024-05-15T12:37:52.324615+0000","last_peered":"2024-05-15T12:37:52.324615+0000","last_clean":"2024-05-15T12:37:52.324615+0000","last_became_active":"2024-05-15T12:37:13.082899+0000","last_became_peered":"2024-05-15T12:37:13.082899+0000","last_unstale":"2024-05-15T12:37:52.324615+0000","last_undegraded":"2024-05-15T12:37:52.324615+0000","last_fullsized":"2024-05-15T12:37:52.324615+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:58:36.819505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320145+0000","last_change":"2024-05-15T12:37:09.069604+0000","last_active":"2024-05-15T12:37:52.320145+0000","last_peered":"2024-05-15T12:37:52.320145+0000","last_clean":"2024-05-15T12:37:52.320145+0000","last_became_active":"2024-05-15T12:37:09.068379+0000","last_became_peered":"2024-05-15T12:37:09.068379+0000","last_unstale":"2024-05-15T12:37:52.320145+0000","last_undegraded":"2024-05-15T12:37:52.320145+0000","last_fullsized":"2024-05-15T12:37:52.320145+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:17:03.808516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428962+0000","last_change":"2024-05-15T12:37:11.082781+0000","last_active":"2024-05-15T12:37:52.428962+0000","last_peered":"2024-05-15T12:37:52.428962+0000","last_clean":"2024-05-15T12:37:52.428962+0000","last_became_active":"2024-05-15T12:37:11.082497+0000","last_became_peered":"2024-05-15T12:37:11.082497+0000","last_unstale":"2024-05-15T12:37:52.428962+0000","last_undegraded":"2024-05-15T12:37:52.428962+0000","last_fullsized":"2024-05-15T12:37:52.428962+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:33:41.512925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545832+0000","last_change":"2024-05-15T12:37:15.316767+0000","last_active":"2024-05-15T12:37:52.545832+0000","last_peered":"2024-05-15T12:37:52.545832+0000","last_clean":"2024-05-15T12:37:52.545832+0000","last_became_active":"2024-05-15T12:37:15.316340+0000","last_became_peered":"2024-05-15T12:37:15.316340+0000","last_unstale":"2024-05-15T12:37:52.545832+0000","last_undegraded":"2024-05-15T12:37:52.545832+0000","last_fullsized":"2024-05-15T12:37:52.545832+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:44:14.714780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323529+0000","last_change":"2024-05-15T12:37:13.092984+0000","last_active":"2024-05-15T12:37:52.323529+0000","last_peered":"2024-05-15T12:37:52.323529+0000","last_clean":"2024-05-15T12:37:52.323529+0000","last_became_active":"2024-05-15T12:37:13.092666+0000","last_became_peered":"2024-05-15T12:37:13.092666+0000","last_unstale":"2024-05-15T12:37:52.323529+0000","last_undegraded":"2024-05-15T12:37:52.323529+0000","last_fullsized":"2024-05-15T12:37:52.323529+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:26:13.925238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323327+0000","last_change":"2024-05-15T12:37:09.066768+0000","last_active":"2024-05-15T12:37:52.323327+0000","last_peered":"2024-05-15T12:37:52.323327+0000","last_clean":"2024-05-15T12:37:52.323327+0000","last_became_active":"2024-05-15T12:37:09.066431+0000","last_became_peered":"2024-05-15T12:37:09.066431+0000","last_unstale":"2024-05-15T12:37:52.323327+0000","last_undegraded":"2024-05-15T12:37:52.323327+0000","last_fullsized":"2024-05-15T12:37:52.323327+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:21:07.246383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545013+0000","last_change":"2024-05-15T12:37:11.087059+0000","last_active":"2024-05-15T12:37:52.545013+0000","last_peered":"2024-05-15T12:37:52.545013+0000","last_clean":"2024-05-15T12:37:52.545013+0000","last_became_active":"2024-05-15T12:37:11.086355+0000","last_became_peered":"2024-05-15T12:37:11.086355+0000","last_unstale":"2024-05-15T12:37:52.545013+0000","last_undegraded":"2024-05-15T12:37:52.545013+0000","last_fullsized":"2024-05-15T12:37:52.545013+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:18:52.297504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":23,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433929+0000","last_change":"2024-05-15T12:37:15.107738+0000","last_active":"2024-05-15T12:37:52.433929+0000","last_peered":"2024-05-15T12:37:52.433929+0000","last_clean":"2024-05-15T12:37:52.433929+0000","last_became_active":"2024-05-15T12:37:15.107357+0000","last_became_peered":"2024-05-15T12:37:15.107357+0000","last_unstale":"2024-05-15T12:37:52.433929+0000","last_undegraded":"2024-05-15T12:37:52.433929+0000","last_fullsized":"2024-05-15T12:37:52.433929+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:34:22.770667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'11","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130583+0000","last_change":"2024-05-15T12:37:13.095415+0000","last_active":"2024-05-15T12:38:34.130583+0000","last_peered":"2024-05-15T12:38:34.130583+0000","last_clean":"2024-05-15T12:38:34.130583+0000","last_became_active":"2024-05-15T12:37:13.095086+0000","last_became_peered":"2024-05-15T12:37:13.095086+0000","last_unstale":"2024-05-15T12:38:34.130583+0000","last_undegraded":"2024-05-15T12:38:34.130583+0000","last_fullsized":"2024-05-15T12:38:34.130583+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:57:59.443236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323718+0000","last_change":"2024-05-15T12:37:09.069433+0000","last_active":"2024-05-15T12:37:52.323718+0000","last_peered":"2024-05-15T12:37:52.323718+0000","last_clean":"2024-05-15T12:37:52.323718+0000","last_became_active":"2024-05-15T12:37:09.068957+0000","last_became_peered":"2024-05-15T12:37:09.068957+0000","last_unstale":"2024-05-15T12:37:52.323718+0000","last_undegraded":"2024-05-15T12:37:52.323718+0000","last_fullsized":"2024-05-15T12:37:52.323718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:20:17.612056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320928+0000","last_change":"2024-05-15T12:37:11.080224+0000","last_active":"2024-05-15T12:37:52.320928+0000","last_peered":"2024-05-15T12:37:52.320928+0000","last_clean":"2024-05-15T12:37:52.320928+0000","last_became_active":"2024-05-15T12:37:11.079961+0000","last_became_peered":"2024-05-15T12:37:11.079961+0000","last_unstale":"2024-05-15T12:37:52.320928+0000","last_undegraded":"2024-05-15T12:37:52.320928+0000","last_fullsized":"2024-05-15T12:37:52.320928+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:56:54.697203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324755+0000","last_change":"2024-05-15T12:37:15.102480+0000","last_active":"2024-05-15T12:37:52.324755+0000","last_peered":"2024-05-15T12:37:52.324755+0000","last_clean":"2024-05-15T12:37:52.324755+0000","last_became_active":"2024-05-15T12:37:15.102154+0000","last_became_peered":"2024-05-15T12:37:15.102154+0000","last_unstale":"2024-05-15T12:37:52.324755+0000","last_undegraded":"2024-05-15T12:37:52.324755+0000","last_fullsized":"2024-05-15T12:37:52.324755+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:31:30.098668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.321070+0000","last_change":"2024-05-15T12:37:13.089935+0000","last_active":"2024-05-15T12:37:52.321070+0000","last_peered":"2024-05-15T12:37:52.321070+0000","last_clean":"2024-05-15T12:37:52.321070+0000","last_became_active":"2024-05-15T12:37:13.089530+0000","last_became_peered":"2024-05-15T12:37:13.089530+0000","last_unstale":"2024-05-15T12:37:52.321070+0000","last_undegraded":"2024-05-15T12:37:52.321070+0000","last_fullsized":"2024-05-15T12:37:52.321070+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:19:19.178879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544154+0000","last_change":"2024-05-15T12:37:09.073841+0000","last_active":"2024-05-15T12:37:52.544154+0000","last_peered":"2024-05-15T12:37:52.544154+0000","last_clean":"2024-05-15T12:37:52.544154+0000","last_became_active":"2024-05-15T12:37:09.073549+0000","last_became_peered":"2024-05-15T12:37:09.073549+0000","last_unstale":"2024-05-15T12:37:52.544154+0000","last_undegraded":"2024-05-15T12:37:52.544154+0000","last_fullsized":"2024-05-15T12:37:52.544154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:31:02.467093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544863+0000","last_change":"2024-05-15T12:37:11.085950+0000","last_active":"2024-05-15T12:37:52.544863+0000","last_peered":"2024-05-15T12:37:52.544863+0000","last_clean":"2024-05-15T12:37:52.544863+0000","last_became_active":"2024-05-15T12:37:11.085631+0000","last_became_peered":"2024-05-15T12:37:11.085631+0000","last_unstale":"2024-05-15T12:37:52.544863+0000","last_undegraded":"2024-05-15T12:37:52.544863+0000","last_fullsized":"2024-05-15T12:37:52.544863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:28:52.396142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433582+0000","last_change":"2024-05-15T12:37:15.106738+0000","last_active":"2024-05-15T12:37:52.433582+0000","last_peered":"2024-05-15T12:37:52.433582+0000","last_clean":"2024-05-15T12:37:52.433582+0000","last_became_active":"2024-05-15T12:37:15.106417+0000","last_became_peered":"2024-05-15T12:37:15.106417+0000","last_unstale":"2024-05-15T12:37:52.433582+0000","last_undegraded":"2024-05-15T12:37:52.433582+0000","last_fullsized":"2024-05-15T12:37:52.433582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:37:20.699673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544251+0000","last_change":"2024-05-15T12:37:13.093826+0000","last_active":"2024-05-15T12:37:52.544251+0000","last_peered":"2024-05-15T12:37:52.544251+0000","last_clean":"2024-05-15T12:37:52.544251+0000","last_became_active":"2024-05-15T12:37:13.093542+0000","last_became_peered":"2024-05-15T12:37:13.093542+0000","last_unstale":"2024-05-15T12:37:52.544251+0000","last_undegraded":"2024-05-15T12:37:52.544251+0000","last_fullsized":"2024-05-15T12:37:52.544251+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:11:10.979708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.321043+0000","last_change":"2024-05-15T12:37:09.070640+0000","last_active":"2024-05-15T12:37:52.321043+0000","last_peered":"2024-05-15T12:37:52.321043+0000","last_clean":"2024-05-15T12:37:52.321043+0000","last_became_active":"2024-05-15T12:37:09.070372+0000","last_became_peered":"2024-05-15T12:37:09.070372+0000","last_unstale":"2024-05-15T12:37:52.321043+0000","last_undegraded":"2024-05-15T12:37:52.321043+0000","last_fullsized":"2024-05-15T12:37:52.321043+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:59:46.603095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544165+0000","last_change":"2024-05-15T12:37:11.088186+0000","last_active":"2024-05-15T12:37:52.544165+0000","last_peered":"2024-05-15T12:37:52.544165+0000","last_clean":"2024-05-15T12:37:52.544165+0000","last_became_active":"2024-05-15T12:37:11.087828+0000","last_became_peered":"2024-05-15T12:37:11.087828+0000","last_unstale":"2024-05-15T12:37:52.544165+0000","last_undegraded":"2024-05-15T12:37:52.544165+0000","last_fullsized":"2024-05-15T12:37:52.544165+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:09:53.788757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544470+0000","last_change":"2024-05-15T12:37:15.319273+0000","last_active":"2024-05-15T12:37:52.544470+0000","last_peered":"2024-05-15T12:37:52.544470+0000","last_clean":"2024-05-15T12:37:52.544470+0000","last_became_active":"2024-05-15T12:37:15.318887+0000","last_became_peered":"2024-05-15T12:37:15.318887+0000","last_unstale":"2024-05-15T12:37:52.544470+0000","last_undegraded":"2024-05-15T12:37:52.544470+0000","last_fullsized":"2024-05-15T12:37:52.544470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:00:51.121558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433718+0000","last_change":"2024-05-15T12:37:13.095074+0000","last_active":"2024-05-15T12:37:52.433718+0000","last_peered":"2024-05-15T12:37:52.433718+0000","last_clean":"2024-05-15T12:37:52.433718+0000","last_became_active":"2024-05-15T12:37:13.094778+0000","last_became_peered":"2024-05-15T12:37:13.094778+0000","last_unstale":"2024-05-15T12:37:52.433718+0000","last_undegraded":"2024-05-15T12:37:52.433718+0000","last_fullsized":"2024-05-15T12:37:52.433718+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:26:08.311809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421017+0000","last_change":"2024-05-15T12:37:09.071500+0000","last_active":"2024-05-15T12:37:52.421017+0000","last_peered":"2024-05-15T12:37:52.421017+0000","last_clean":"2024-05-15T12:37:52.421017+0000","last_became_active":"2024-05-15T12:37:09.071211+0000","last_became_peered":"2024-05-15T12:37:09.071211+0000","last_unstale":"2024-05-15T12:37:52.421017+0000","last_undegraded":"2024-05-15T12:37:52.421017+0000","last_fullsized":"2024-05-15T12:37:52.421017+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:02:16.995090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544384+0000","last_change":"2024-05-15T12:37:11.084426+0000","last_active":"2024-05-15T12:37:52.544384+0000","last_peered":"2024-05-15T12:37:52.544384+0000","last_clean":"2024-05-15T12:37:52.544384+0000","last_became_active":"2024-05-15T12:37:11.084110+0000","last_became_peered":"2024-05-15T12:37:11.084110+0000","last_unstale":"2024-05-15T12:37:52.544384+0000","last_undegraded":"2024-05-15T12:37:52.544384+0000","last_fullsized":"2024-05-15T12:37:52.544384+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:58:54.713264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319498+0000","last_change":"2024-05-15T12:37:15.105516+0000","last_active":"2024-05-15T12:37:52.319498+0000","last_peered":"2024-05-15T12:37:52.319498+0000","last_clean":"2024-05-15T12:37:52.319498+0000","last_became_active":"2024-05-15T12:37:15.105201+0000","last_became_peered":"2024-05-15T12:37:15.105201+0000","last_unstale":"2024-05-15T12:37:52.319498+0000","last_undegraded":"2024-05-15T12:37:52.319498+0000","last_fullsized":"2024-05-15T12:37:52.319498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:12:33.897811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544763+0000","last_change":"2024-05-15T12:37:13.090467+0000","last_active":"2024-05-15T12:37:52.544763+0000","last_peered":"2024-05-15T12:37:52.544763+0000","last_clean":"2024-05-15T12:37:52.544763+0000","last_became_active":"2024-05-15T12:37:13.090091+0000","last_became_peered":"2024-05-15T12:37:13.090091+0000","last_unstale":"2024-05-15T12:37:52.544763+0000","last_undegraded":"2024-05-15T12:37:52.544763+0000","last_fullsized":"2024-05-15T12:37:52.544763+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:21:25.235934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433998+0000","last_change":"2024-05-15T12:37:09.072613+0000","last_active":"2024-05-15T12:37:52.433998+0000","last_peered":"2024-05-15T12:37:52.433998+0000","last_clean":"2024-05-15T12:37:52.433998+0000","last_became_active":"2024-05-15T12:37:09.072271+0000","last_became_peered":"2024-05-15T12:37:09.072271+0000","last_unstale":"2024-05-15T12:37:52.433998+0000","last_undegraded":"2024-05-15T12:37:52.433998+0000","last_fullsized":"2024-05-15T12:37:52.433998+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:11:26.173203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324100+0000","last_change":"2024-05-15T12:37:11.081801+0000","last_active":"2024-05-15T12:37:52.324100+0000","last_peered":"2024-05-15T12:37:52.324100+0000","last_clean":"2024-05-15T12:37:52.324100+0000","last_became_active":"2024-05-15T12:37:11.080039+0000","last_became_peered":"2024-05-15T12:37:11.080039+0000","last_unstale":"2024-05-15T12:37:52.324100+0000","last_undegraded":"2024-05-15T12:37:52.324100+0000","last_fullsized":"2024-05-15T12:37:52.324100+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:54:56.958666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544154+0000","last_change":"2024-05-15T12:37:15.103262+0000","last_active":"2024-05-15T12:37:52.544154+0000","last_peered":"2024-05-15T12:37:52.544154+0000","last_clean":"2024-05-15T12:37:52.544154+0000","last_became_active":"2024-05-15T12:37:15.102965+0000","last_became_peered":"2024-05-15T12:37:15.102965+0000","last_unstale":"2024-05-15T12:37:52.544154+0000","last_undegraded":"2024-05-15T12:37:52.544154+0000","last_fullsized":"2024-05-15T12:37:52.544154+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:31:24.103174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324003+0000","last_change":"2024-05-15T12:37:13.093157+0000","last_active":"2024-05-15T12:37:52.324003+0000","last_peered":"2024-05-15T12:37:52.324003+0000","last_clean":"2024-05-15T12:37:52.324003+0000","last_became_active":"2024-05-15T12:37:13.092802+0000","last_became_peered":"2024-05-15T12:37:13.092802+0000","last_unstale":"2024-05-15T12:37:52.324003+0000","last_undegraded":"2024-05-15T12:37:52.324003+0000","last_fullsized":"2024-05-15T12:37:52.324003+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:10:42.928759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323851+0000","last_change":"2024-05-15T12:37:09.072446+0000","last_active":"2024-05-15T12:37:52.323851+0000","last_peered":"2024-05-15T12:37:52.323851+0000","last_clean":"2024-05-15T12:37:52.323851+0000","last_became_active":"2024-05-15T12:37:09.072109+0000","last_became_peered":"2024-05-15T12:37:09.072109+0000","last_unstale":"2024-05-15T12:37:52.323851+0000","last_undegraded":"2024-05-15T12:37:52.323851+0000","last_fullsized":"2024-05-15T12:37:52.323851+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:30:07.904230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323968+0000","last_change":"2024-05-15T12:37:24.684617+0000","last_active":"2024-05-15T12:37:52.323968+0000","last_peered":"2024-05-15T12:37:52.323968+0000","last_clean":"2024-05-15T12:37:52.323968+0000","last_became_active":"2024-05-15T12:37:24.668904+0000","last_became_peered":"2024-05-15T12:37:24.668904+0000","last_unstale":"2024-05-15T12:37:52.323968+0000","last_undegraded":"2024-05-15T12:37:52.323968+0000","last_fullsized":"2024-05-15T12:37:52.323968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:14:37.439983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320676+0000","last_change":"2024-05-15T12:37:15.101675+0000","last_active":"2024-05-15T12:37:52.320676+0000","last_peered":"2024-05-15T12:37:52.320676+0000","last_clean":"2024-05-15T12:37:52.320676+0000","last_became_active":"2024-05-15T12:37:15.101362+0000","last_became_peered":"2024-05-15T12:37:15.101362+0000","last_unstale":"2024-05-15T12:37:52.320676+0000","last_undegraded":"2024-05-15T12:37:52.320676+0000","last_fullsized":"2024-05-15T12:37:52.320676+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:43:18.765531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'11","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130583+0000","last_change":"2024-05-15T12:37:13.095655+0000","last_active":"2024-05-15T12:38:34.130583+0000","last_peered":"2024-05-15T12:38:34.130583+0000","last_clean":"2024-05-15T12:38:34.130583+0000","last_became_active":"2024-05-15T12:37:13.095361+0000","last_became_peered":"2024-05-15T12:37:13.095361+0000","last_unstale":"2024-05-15T12:38:34.130583+0000","last_undegraded":"2024-05-15T12:38:34.130583+0000","last_fullsized":"2024-05-15T12:38:34.130583+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:47:10.369343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433787+0000","last_change":"2024-05-15T12:37:09.072627+0000","last_active":"2024-05-15T12:37:52.433787+0000","last_peered":"2024-05-15T12:37:52.433787+0000","last_clean":"2024-05-15T12:37:52.433787+0000","last_became_active":"2024-05-15T12:37:09.072308+0000","last_became_peered":"2024-05-15T12:37:09.072308+0000","last_unstale":"2024-05-15T12:37:52.433787+0000","last_undegraded":"2024-05-15T12:37:52.433787+0000","last_fullsized":"2024-05-15T12:37:52.433787+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:26:24.723885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545054+0000","last_change":"2024-05-15T12:37:11.089554+0000","last_active":"2024-05-15T12:37:52.545054+0000","last_peered":"2024-05-15T12:37:52.545054+0000","last_clean":"2024-05-15T12:37:52.545054+0000","last_became_active":"2024-05-15T12:37:11.088650+0000","last_became_peered":"2024-05-15T12:37:11.088650+0000","last_unstale":"2024-05-15T12:37:52.545054+0000","last_undegraded":"2024-05-15T12:37:52.545054+0000","last_fullsized":"2024-05-15T12:37:52.545054+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:18:02.844729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324342+0000","last_change":"2024-05-15T12:37:15.103999+0000","last_active":"2024-05-15T12:37:52.324342+0000","last_peered":"2024-05-15T12:37:52.324342+0000","last_clean":"2024-05-15T12:37:52.324342+0000","last_became_active":"2024-05-15T12:37:15.103613+0000","last_became_peered":"2024-05-15T12:37:15.103613+0000","last_unstale":"2024-05-15T12:37:52.324342+0000","last_undegraded":"2024-05-15T12:37:52.324342+0000","last_fullsized":"2024-05-15T12:37:52.324342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:12:04.291749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'11","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130531+0000","last_change":"2024-05-15T12:37:13.092841+0000","last_active":"2024-05-15T12:38:34.130531+0000","last_peered":"2024-05-15T12:38:34.130531+0000","last_clean":"2024-05-15T12:38:34.130531+0000","last_became_active":"2024-05-15T12:37:13.092552+0000","last_became_peered":"2024-05-15T12:37:13.092552+0000","last_unstale":"2024-05-15T12:38:34.130531+0000","last_undegraded":"2024-05-15T12:38:34.130531+0000","last_fullsized":"2024-05-15T12:38:34.130531+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:39:13.880161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320046+0000","last_change":"2024-05-15T12:37:09.067840+0000","last_active":"2024-05-15T12:37:52.320046+0000","last_peered":"2024-05-15T12:37:52.320046+0000","last_clean":"2024-05-15T12:37:52.320046+0000","last_became_active":"2024-05-15T12:37:09.067100+0000","last_became_peered":"2024-05-15T12:37:09.067100+0000","last_unstale":"2024-05-15T12:37:52.320046+0000","last_undegraded":"2024-05-15T12:37:52.320046+0000","last_fullsized":"2024-05-15T12:37:52.320046+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:16:07.291371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421185+0000","last_change":"2024-05-15T12:37:11.084780+0000","last_active":"2024-05-15T12:37:52.421185+0000","last_peered":"2024-05-15T12:37:52.421185+0000","last_clean":"2024-05-15T12:37:52.421185+0000","last_became_active":"2024-05-15T12:37:11.084439+0000","last_became_peered":"2024-05-15T12:37:11.084439+0000","last_unstale":"2024-05-15T12:37:52.421185+0000","last_undegraded":"2024-05-15T12:37:52.421185+0000","last_fullsized":"2024-05-15T12:37:52.421185+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:38:37.629599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.429198+0000","last_change":"2024-05-15T12:37:15.107416+0000","last_active":"2024-05-15T12:37:52.429198+0000","last_peered":"2024-05-15T12:37:52.429198+0000","last_clean":"2024-05-15T12:37:52.429198+0000","last_became_active":"2024-05-15T12:37:15.107084+0000","last_became_peered":"2024-05-15T12:37:15.107084+0000","last_unstale":"2024-05-15T12:37:52.429198+0000","last_undegraded":"2024-05-15T12:37:52.429198+0000","last_fullsized":"2024-05-15T12:37:52.429198+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:23:27.889778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545779+0000","last_change":"2024-05-15T12:37:13.091053+0000","last_active":"2024-05-15T12:37:52.545779+0000","last_peered":"2024-05-15T12:37:52.545779+0000","last_clean":"2024-05-15T12:37:52.545779+0000","last_became_active":"2024-05-15T12:37:13.090720+0000","last_became_peered":"2024-05-15T12:37:13.090720+0000","last_unstale":"2024-05-15T12:37:52.545779+0000","last_undegraded":"2024-05-15T12:37:52.545779+0000","last_fullsized":"2024-05-15T12:37:52.545779+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:36:48.582844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"60'1","reported_seq":41,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433569+0000","last_change":"2024-05-15T12:37:09.073488+0000","last_active":"2024-05-15T12:37:52.433569+0000","last_peered":"2024-05-15T12:37:52.433569+0000","last_clean":"2024-05-15T12:37:52.433569+0000","last_became_active":"2024-05-15T12:37:09.073181+0000","last_became_peered":"2024-05-15T12:37:09.073181+0000","last_unstale":"2024-05-15T12:37:52.433569+0000","last_undegraded":"2024-05-15T12:37:52.433569+0000","last_fullsized":"2024-05-15T12:37:52.433569+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:32:47.183558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320933+0000","last_change":"2024-05-15T12:37:11.080880+0000","last_active":"2024-05-15T12:37:52.320933+0000","last_peered":"2024-05-15T12:37:52.320933+0000","last_clean":"2024-05-15T12:37:52.320933+0000","last_became_active":"2024-05-15T12:37:11.080611+0000","last_became_peered":"2024-05-15T12:37:11.080611+0000","last_unstale":"2024-05-15T12:37:52.320933+0000","last_undegraded":"2024-05-15T12:37:52.320933+0000","last_fullsized":"2024-05-15T12:37:52.320933+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:39:59.310154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433982+0000","last_change":"2024-05-15T12:37:15.318629+0000","last_active":"2024-05-15T12:37:52.433982+0000","last_peered":"2024-05-15T12:37:52.433982+0000","last_clean":"2024-05-15T12:37:52.433982+0000","last_became_active":"2024-05-15T12:37:15.318289+0000","last_became_peered":"2024-05-15T12:37:15.318289+0000","last_unstale":"2024-05-15T12:37:52.433982+0000","last_undegraded":"2024-05-15T12:37:52.433982+0000","last_fullsized":"2024-05-15T12:37:52.433982+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:41:50.527836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421051+0000","last_change":"2024-05-15T12:37:13.096162+0000","last_active":"2024-05-15T12:37:52.421051+0000","last_peered":"2024-05-15T12:37:52.421051+0000","last_clean":"2024-05-15T12:37:52.421051+0000","last_became_active":"2024-05-15T12:37:13.095884+0000","last_became_peered":"2024-05-15T12:37:13.095884+0000","last_unstale":"2024-05-15T12:37:52.421051+0000","last_undegraded":"2024-05-15T12:37:52.421051+0000","last_fullsized":"2024-05-15T12:37:52.421051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:27:10.590238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544243+0000","last_change":"2024-05-15T12:37:09.068210+0000","last_active":"2024-05-15T12:37:52.544243+0000","last_peered":"2024-05-15T12:37:52.544243+0000","last_clean":"2024-05-15T12:37:52.544243+0000","last_became_active":"2024-05-15T12:37:09.067827+0000","last_became_peered":"2024-05-15T12:37:09.067827+0000","last_unstale":"2024-05-15T12:37:52.544243+0000","last_undegraded":"2024-05-15T12:37:52.544243+0000","last_fullsized":"2024-05-15T12:37:52.544243+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:30:55.087299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545877+0000","last_change":"2024-05-15T12:37:11.089151+0000","last_active":"2024-05-15T12:37:52.545877+0000","last_peered":"2024-05-15T12:37:52.545877+0000","last_clean":"2024-05-15T12:37:52.545877+0000","last_became_active":"2024-05-15T12:37:11.088858+0000","last_became_peered":"2024-05-15T12:37:11.088858+0000","last_unstale":"2024-05-15T12:37:52.545877+0000","last_undegraded":"2024-05-15T12:37:52.545877+0000","last_fullsized":"2024-05-15T12:37:52.545877+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:02:51.858503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420784+0000","last_change":"2024-05-15T12:37:15.316724+0000","last_active":"2024-05-15T12:37:52.420784+0000","last_peered":"2024-05-15T12:37:52.420784+0000","last_clean":"2024-05-15T12:37:52.420784+0000","last_became_active":"2024-05-15T12:37:15.316391+0000","last_became_peered":"2024-05-15T12:37:15.316391+0000","last_unstale":"2024-05-15T12:37:52.420784+0000","last_undegraded":"2024-05-15T12:37:52.420784+0000","last_fullsized":"2024-05-15T12:37:52.420784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:53:57.229038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434345+0000","last_change":"2024-05-15T12:37:09.074136+0000","last_active":"2024-05-15T12:37:52.434345+0000","last_peered":"2024-05-15T12:37:52.434345+0000","last_clean":"2024-05-15T12:37:52.434345+0000","last_became_active":"2024-05-15T12:37:09.073851+0000","last_became_peered":"2024-05-15T12:37:09.073851+0000","last_unstale":"2024-05-15T12:37:52.434345+0000","last_undegraded":"2024-05-15T12:37:52.434345+0000","last_fullsized":"2024-05-15T12:37:52.434345+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T21:29:57.880157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420550+0000","last_change":"2024-05-15T12:37:11.084555+0000","last_active":"2024-05-15T12:37:52.420550+0000","last_peered":"2024-05-15T12:37:52.420550+0000","last_clean":"2024-05-15T12:37:52.420550+0000","last_became_active":"2024-05-15T12:37:11.084182+0000","last_became_peered":"2024-05-15T12:37:11.084182+0000","last_unstale":"2024-05-15T12:37:52.420550+0000","last_undegraded":"2024-05-15T12:37:52.420550+0000","last_fullsized":"2024-05-15T12:37:52.420550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:07:24.527454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'11","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130585+0000","last_change":"2024-05-15T12:37:13.094232+0000","last_active":"2024-05-15T12:38:34.130585+0000","last_peered":"2024-05-15T12:38:34.130585+0000","last_clean":"2024-05-15T12:38:34.130585+0000","last_became_active":"2024-05-15T12:37:13.093938+0000","last_became_peered":"2024-05-15T12:37:13.093938+0000","last_unstale":"2024-05-15T12:38:34.130585+0000","last_undegraded":"2024-05-15T12:38:34.130585+0000","last_fullsized":"2024-05-15T12:38:34.130585+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:01:05.717989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544574+0000","last_change":"2024-05-15T12:37:15.109119+0000","last_active":"2024-05-15T12:37:52.544574+0000","last_peered":"2024-05-15T12:37:52.544574+0000","last_clean":"2024-05-15T12:37:52.544574+0000","last_became_active":"2024-05-15T12:37:15.108741+0000","last_became_peered":"2024-05-15T12:37:15.108741+0000","last_unstale":"2024-05-15T12:37:52.544574+0000","last_undegraded":"2024-05-15T12:37:52.544574+0000","last_fullsized":"2024-05-15T12:37:52.544574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:57:18.964707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434185+0000","last_change":"2024-05-15T12:37:09.072892+0000","last_active":"2024-05-15T12:37:52.434185+0000","last_peered":"2024-05-15T12:37:52.434185+0000","last_clean":"2024-05-15T12:37:52.434185+0000","last_became_active":"2024-05-15T12:37:09.072354+0000","last_became_peered":"2024-05-15T12:37:09.072354+0000","last_unstale":"2024-05-15T12:37:52.434185+0000","last_undegraded":"2024-05-15T12:37:52.434185+0000","last_fullsized":"2024-05-15T12:37:52.434185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:35:29.903053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323305+0000","last_change":"2024-05-15T12:37:11.092438+0000","last_active":"2024-05-15T12:37:52.323305+0000","last_peered":"2024-05-15T12:37:52.323305+0000","last_clean":"2024-05-15T12:37:52.323305+0000","last_became_active":"2024-05-15T12:37:11.092120+0000","last_became_peered":"2024-05-15T12:37:11.092120+0000","last_unstale":"2024-05-15T12:37:52.323305+0000","last_undegraded":"2024-05-15T12:37:52.323305+0000","last_fullsized":"2024-05-15T12:37:52.323305+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:16:27.383512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428364+0000","last_change":"2024-05-15T12:37:13.095561+0000","last_active":"2024-05-15T12:37:52.428364+0000","last_peered":"2024-05-15T12:37:52.428364+0000","last_clean":"2024-05-15T12:37:52.428364+0000","last_became_active":"2024-05-15T12:37:13.095239+0000","last_became_peered":"2024-05-15T12:37:13.095239+0000","last_unstale":"2024-05-15T12:37:52.428364+0000","last_undegraded":"2024-05-15T12:37:52.428364+0000","last_fullsized":"2024-05-15T12:37:52.428364+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:31:54.333216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433447+0000","last_change":"2024-05-15T12:37:15.108996+0000","last_active":"2024-05-15T12:37:52.433447+0000","last_peered":"2024-05-15T12:37:52.433447+0000","last_clean":"2024-05-15T12:37:52.433447+0000","last_became_active":"2024-05-15T12:37:15.108262+0000","last_became_peered":"2024-05-15T12:37:15.108262+0000","last_unstale":"2024-05-15T12:37:52.433447+0000","last_undegraded":"2024-05-15T12:37:52.433447+0000","last_fullsized":"2024-05-15T12:37:52.433447+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:31:52.319671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428864+0000","last_change":"2024-05-15T12:37:13.096141+0000","last_active":"2024-05-15T12:37:52.428864+0000","last_peered":"2024-05-15T12:37:52.428864+0000","last_clean":"2024-05-15T12:37:52.428864+0000","last_became_active":"2024-05-15T12:37:13.095772+0000","last_became_peered":"2024-05-15T12:37:13.095772+0000","last_unstale":"2024-05-15T12:37:52.428864+0000","last_undegraded":"2024-05-15T12:37:52.428864+0000","last_fullsized":"2024-05-15T12:37:52.428864+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:16:40.330004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434243+0000","last_change":"2024-05-15T12:37:09.069356+0000","last_active":"2024-05-15T12:37:52.434243+0000","last_peered":"2024-05-15T12:37:52.434243+0000","last_clean":"2024-05-15T12:37:52.434243+0000","last_became_active":"2024-05-15T12:37:09.069033+0000","last_became_peered":"2024-05-15T12:37:09.069033+0000","last_unstale":"2024-05-15T12:37:52.434243+0000","last_undegraded":"2024-05-15T12:37:52.434243+0000","last_fullsized":"2024-05-15T12:37:52.434243+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:35:33.517568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544605+0000","last_change":"2024-05-15T12:37:11.088397+0000","last_active":"2024-05-15T12:37:52.544605+0000","last_peered":"2024-05-15T12:37:52.544605+0000","last_clean":"2024-05-15T12:37:52.544605+0000","last_became_active":"2024-05-15T12:37:11.088067+0000","last_became_peered":"2024-05-15T12:37:11.088067+0000","last_unstale":"2024-05-15T12:37:52.544605+0000","last_undegraded":"2024-05-15T12:37:52.544605+0000","last_fullsized":"2024-05-15T12:37:52.544605+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:54:34.590074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319568+0000","last_change":"2024-05-15T12:37:15.107405+0000","last_active":"2024-05-15T12:37:52.319568+0000","last_peered":"2024-05-15T12:37:52.319568+0000","last_clean":"2024-05-15T12:37:52.319568+0000","last_became_active":"2024-05-15T12:37:15.107101+0000","last_became_peered":"2024-05-15T12:37:15.107101+0000","last_unstale":"2024-05-15T12:37:52.319568+0000","last_undegraded":"2024-05-15T12:37:52.319568+0000","last_fullsized":"2024-05-15T12:37:52.319568+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:04:19.513224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428908+0000","last_change":"2024-05-15T12:37:13.090322+0000","last_active":"2024-05-15T12:37:52.428908+0000","last_peered":"2024-05-15T12:37:52.428908+0000","last_clean":"2024-05-15T12:37:52.428908+0000","last_became_active":"2024-05-15T12:37:13.090014+0000","last_became_peered":"2024-05-15T12:37:13.090014+0000","last_unstale":"2024-05-15T12:37:52.428908+0000","last_undegraded":"2024-05-15T12:37:52.428908+0000","last_fullsized":"2024-05-15T12:37:52.428908+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:44:24.689515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420820+0000","last_change":"2024-05-15T12:37:09.068510+0000","last_active":"2024-05-15T12:37:52.420820+0000","last_peered":"2024-05-15T12:37:52.420820+0000","last_clean":"2024-05-15T12:37:52.420820+0000","last_became_active":"2024-05-15T12:37:09.068206+0000","last_became_peered":"2024-05-15T12:37:09.068206+0000","last_unstale":"2024-05-15T12:37:52.420820+0000","last_undegraded":"2024-05-15T12:37:52.420820+0000","last_fullsized":"2024-05-15T12:37:52.420820+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:39:56.798950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320297+0000","last_change":"2024-05-15T12:37:11.084962+0000","last_active":"2024-05-15T12:37:52.320297+0000","last_peered":"2024-05-15T12:37:52.320297+0000","last_clean":"2024-05-15T12:37:52.320297+0000","last_became_active":"2024-05-15T12:37:11.084648+0000","last_became_peered":"2024-05-15T12:37:11.084648+0000","last_unstale":"2024-05-15T12:37:52.320297+0000","last_undegraded":"2024-05-15T12:37:52.320297+0000","last_fullsized":"2024-05-15T12:37:52.320297+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:30:23.386442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545312+0000","last_change":"2024-05-15T12:37:15.107641+0000","last_active":"2024-05-15T12:37:52.545312+0000","last_peered":"2024-05-15T12:37:52.545312+0000","last_clean":"2024-05-15T12:37:52.545312+0000","last_became_active":"2024-05-15T12:37:15.107336+0000","last_became_peered":"2024-05-15T12:37:15.107336+0000","last_unstale":"2024-05-15T12:37:52.545312+0000","last_undegraded":"2024-05-15T12:37:52.545312+0000","last_fullsized":"2024-05-15T12:37:52.545312+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:55:52.656745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420602+0000","last_change":"2024-05-15T12:37:13.091923+0000","last_active":"2024-05-15T12:37:52.420602+0000","last_peered":"2024-05-15T12:37:52.420602+0000","last_clean":"2024-05-15T12:37:52.420602+0000","last_became_active":"2024-05-15T12:37:09.071579+0000","last_became_peered":"2024-05-15T12:37:09.071579+0000","last_unstale":"2024-05-15T12:37:52.420602+0000","last_undegraded":"2024-05-15T12:37:52.420602+0000","last_fullsized":"2024-05-15T12:37:52.420602+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:03:11.283684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049982009999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323592+0000","last_change":"2024-05-15T12:37:09.065661+0000","last_active":"2024-05-15T12:37:52.323592+0000","last_peered":"2024-05-15T12:37:52.323592+0000","last_clean":"2024-05-15T12:37:52.323592+0000","last_became_active":"2024-05-15T12:37:09.065283+0000","last_became_peered":"2024-05-15T12:37:09.065283+0000","last_unstale":"2024-05-15T12:37:52.323592+0000","last_undegraded":"2024-05-15T12:37:52.323592+0000","last_fullsized":"2024-05-15T12:37:52.323592+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:51:06.229965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'30","reported_seq":99,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130531+0000","last_change":"2024-05-15T12:37:11.083252+0000","last_active":"2024-05-15T12:38:34.130531+0000","last_peered":"2024-05-15T12:38:34.130531+0000","last_clean":"2024-05-15T12:38:34.130531+0000","last_became_active":"2024-05-15T12:37:11.082559+0000","last_became_peered":"2024-05-15T12:37:11.082559+0000","last_unstale":"2024-05-15T12:38:34.130531+0000","last_undegraded":"2024-05-15T12:38:34.130531+0000","last_fullsized":"2024-05-15T12:38:34.130531+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:27:09.894865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320646+0000","last_change":"2024-05-15T12:37:13.083190+0000","last_active":"2024-05-15T12:37:52.320646+0000","last_peered":"2024-05-15T12:37:52.320646+0000","last_clean":"2024-05-15T12:37:52.320646+0000","last_became_active":"2024-05-15T12:37:13.082778+0000","last_became_peered":"2024-05-15T12:37:13.082778+0000","last_unstale":"2024-05-15T12:37:52.320646+0000","last_undegraded":"2024-05-15T12:37:52.320646+0000","last_fullsized":"2024-05-15T12:37:52.320646+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:43:52.008364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.434354+0000","last_change":"2024-05-15T12:37:15.317087+0000","last_active":"2024-05-15T12:37:52.434354+0000","last_peered":"2024-05-15T12:37:52.434354+0000","last_clean":"2024-05-15T12:37:52.434354+0000","last_became_active":"2024-05-15T12:37:15.316784+0000","last_became_peered":"2024-05-15T12:37:15.316784+0000","last_unstale":"2024-05-15T12:37:52.434354+0000","last_undegraded":"2024-05-15T12:37:52.434354+0000","last_fullsized":"2024-05-15T12:37:52.434354+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:27:52.133545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323641+0000","last_change":"2024-05-15T12:37:09.070865+0000","last_active":"2024-05-15T12:37:52.323641+0000","last_peered":"2024-05-15T12:37:52.323641+0000","last_clean":"2024-05-15T12:37:52.323641+0000","last_became_active":"2024-05-15T12:37:09.070420+0000","last_became_peered":"2024-05-15T12:37:09.070420+0000","last_unstale":"2024-05-15T12:37:52.323641+0000","last_undegraded":"2024-05-15T12:37:52.323641+0000","last_fullsized":"2024-05-15T12:37:52.323641+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:24:39.138873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428585+0000","last_change":"2024-05-15T12:37:13.086687+0000","last_active":"2024-05-15T12:37:52.428585+0000","last_peered":"2024-05-15T12:37:52.428585+0000","last_clean":"2024-05-15T12:37:52.428585+0000","last_became_active":"2024-05-15T12:37:09.063201+0000","last_became_peered":"2024-05-15T12:37:09.063201+0000","last_unstale":"2024-05-15T12:37:52.428585+0000","last_undegraded":"2024-05-15T12:37:52.428585+0000","last_fullsized":"2024-05-15T12:37:52.428585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:21:46.175922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0004594,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.325360+0000","last_change":"2024-05-15T12:37:11.079670+0000","last_active":"2024-05-15T12:37:52.325360+0000","last_peered":"2024-05-15T12:37:52.325360+0000","last_clean":"2024-05-15T12:37:52.325360+0000","last_became_active":"2024-05-15T12:37:11.078783+0000","last_became_peered":"2024-05-15T12:37:11.078783+0000","last_unstale":"2024-05-15T12:37:52.325360+0000","last_undegraded":"2024-05-15T12:37:52.325360+0000","last_fullsized":"2024-05-15T12:37:52.325360+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:28:21.665496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428933+0000","last_change":"2024-05-15T12:37:13.091820+0000","last_active":"2024-05-15T12:37:52.428933+0000","last_peered":"2024-05-15T12:37:52.428933+0000","last_clean":"2024-05-15T12:37:52.428933+0000","last_became_active":"2024-05-15T12:37:13.091518+0000","last_became_peered":"2024-05-15T12:37:13.091518+0000","last_unstale":"2024-05-15T12:37:52.428933+0000","last_undegraded":"2024-05-15T12:37:52.428933+0000","last_fullsized":"2024-05-15T12:37:52.428933+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:22:05.504481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421136+0000","last_change":"2024-05-15T12:37:15.104655+0000","last_active":"2024-05-15T12:37:52.421136+0000","last_peered":"2024-05-15T12:37:52.421136+0000","last_clean":"2024-05-15T12:37:52.421136+0000","last_became_active":"2024-05-15T12:37:15.104355+0000","last_became_peered":"2024-05-15T12:37:15.104355+0000","last_unstale":"2024-05-15T12:37:52.421136+0000","last_undegraded":"2024-05-15T12:37:52.421136+0000","last_fullsized":"2024-05-15T12:37:52.421136+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:24:33.872619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":345,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:54.416697+0000","last_change":"2024-05-15T12:36:59.870964+0000","last_active":"2024-05-15T12:37:54.416697+0000","last_peered":"2024-05-15T12:37:54.416697+0000","last_clean":"2024-05-15T12:37:54.416697+0000","last_became_active":"2024-05-15T12:36:59.870462+0000","last_became_peered":"2024-05-15T12:36:59.870462+0000","last_unstale":"2024-05-15T12:37:54.416697+0000","last_undegraded":"2024-05-15T12:37:54.416697+0000","last_fullsized":"2024-05-15T12:37:54.416697+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:34:45.857437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:34:45.857437+0000","last_clean_scrub_stamp":"2024-05-15T12:34:45.857437+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:41:02.759031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544666+0000","last_change":"2024-05-15T12:37:09.070440+0000","last_active":"2024-05-15T12:37:52.544666+0000","last_peered":"2024-05-15T12:37:52.544666+0000","last_clean":"2024-05-15T12:37:52.544666+0000","last_became_active":"2024-05-15T12:37:09.069855+0000","last_became_peered":"2024-05-15T12:37:09.069855+0000","last_unstale":"2024-05-15T12:37:52.544666+0000","last_undegraded":"2024-05-15T12:37:52.544666+0000","last_fullsized":"2024-05-15T12:37:52.544666+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:26:00.544976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":70,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130861+0000","last_change":"2024-05-15T12:37:11.084395+0000","last_active":"2024-05-15T12:38:34.130861+0000","last_peered":"2024-05-15T12:38:34.130861+0000","last_clean":"2024-05-15T12:38:34.130861+0000","last_became_active":"2024-05-15T12:37:11.084083+0000","last_became_peered":"2024-05-15T12:37:11.084083+0000","last_unstale":"2024-05-15T12:38:34.130861+0000","last_undegraded":"2024-05-15T12:38:34.130861+0000","last_fullsized":"2024-05-15T12:38:34.130861+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:09:58.669195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433993+0000","last_change":"2024-05-15T12:37:13.100042+0000","last_active":"2024-05-15T12:37:52.433993+0000","last_peered":"2024-05-15T12:37:52.433993+0000","last_clean":"2024-05-15T12:37:52.433993+0000","last_became_active":"2024-05-15T12:37:13.099799+0000","last_became_peered":"2024-05-15T12:37:13.099799+0000","last_unstale":"2024-05-15T12:37:52.433993+0000","last_undegraded":"2024-05-15T12:37:52.433993+0000","last_fullsized":"2024-05-15T12:37:52.433993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:26:02.891879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324553+0000","last_change":"2024-05-15T12:37:15.103262+0000","last_active":"2024-05-15T12:37:52.324553+0000","last_peered":"2024-05-15T12:37:52.324553+0000","last_clean":"2024-05-15T12:37:52.324553+0000","last_became_active":"2024-05-15T12:37:15.102974+0000","last_became_peered":"2024-05-15T12:37:15.102974+0000","last_unstale":"2024-05-15T12:37:52.324553+0000","last_undegraded":"2024-05-15T12:37:52.324553+0000","last_fullsized":"2024-05-15T12:37:52.324553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:55:44.242107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319701+0000","last_change":"2024-05-15T12:37:09.066707+0000","last_active":"2024-05-15T12:37:52.319701+0000","last_peered":"2024-05-15T12:37:52.319701+0000","last_clean":"2024-05-15T12:37:52.319701+0000","last_became_active":"2024-05-15T12:37:09.066314+0000","last_became_peered":"2024-05-15T12:37:09.066314+0000","last_unstale":"2024-05-15T12:37:52.319701+0000","last_undegraded":"2024-05-15T12:37:52.319701+0000","last_fullsized":"2024-05-15T12:37:52.319701+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T16:00:57.682654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":130,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:41.892614+0000","last_change":"2024-05-15T12:37:13.088457+0000","last_active":"2024-05-15T12:38:41.892614+0000","last_peered":"2024-05-15T12:38:41.892614+0000","last_clean":"2024-05-15T12:38:41.892614+0000","last_became_active":"2024-05-15T12:37:09.074657+0000","last_became_peered":"2024-05-15T12:37:09.074657+0000","last_unstale":"2024-05-15T12:38:41.892614+0000","last_undegraded":"2024-05-15T12:38:41.892614+0000","last_fullsized":"2024-05-15T12:38:41.892614+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:05:56.605060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080141299999999995,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":87,"num_read_kb":82,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320770+0000","last_change":"2024-05-15T12:37:11.077982+0000","last_active":"2024-05-15T12:37:52.320770+0000","last_peered":"2024-05-15T12:37:52.320770+0000","last_clean":"2024-05-15T12:37:52.320770+0000","last_became_active":"2024-05-15T12:37:11.077659+0000","last_became_peered":"2024-05-15T12:37:11.077659+0000","last_unstale":"2024-05-15T12:37:52.320770+0000","last_undegraded":"2024-05-15T12:37:52.320770+0000","last_fullsized":"2024-05-15T12:37:52.320770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:25:39.482645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421258+0000","last_change":"2024-05-15T12:37:13.096395+0000","last_active":"2024-05-15T12:37:52.421258+0000","last_peered":"2024-05-15T12:37:52.421258+0000","last_clean":"2024-05-15T12:37:52.421258+0000","last_became_active":"2024-05-15T12:37:13.096105+0000","last_became_peered":"2024-05-15T12:37:13.096105+0000","last_unstale":"2024-05-15T12:37:52.421258+0000","last_undegraded":"2024-05-15T12:37:52.421258+0000","last_fullsized":"2024-05-15T12:37:52.421258+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:18:11.570497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323475+0000","last_change":"2024-05-15T12:37:15.319169+0000","last_active":"2024-05-15T12:37:52.323475+0000","last_peered":"2024-05-15T12:37:52.323475+0000","last_clean":"2024-05-15T12:37:52.323475+0000","last_became_active":"2024-05-15T12:37:15.318837+0000","last_became_peered":"2024-05-15T12:37:15.318837+0000","last_unstale":"2024-05-15T12:37:52.323475+0000","last_undegraded":"2024-05-15T12:37:52.323475+0000","last_fullsized":"2024-05-15T12:37:52.323475+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:53:26.379829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324783+0000","last_change":"2024-05-15T12:37:09.072210+0000","last_active":"2024-05-15T12:37:52.324783+0000","last_peered":"2024-05-15T12:37:52.324783+0000","last_clean":"2024-05-15T12:37:52.324783+0000","last_became_active":"2024-05-15T12:37:09.071893+0000","last_became_peered":"2024-05-15T12:37:09.071893+0000","last_unstale":"2024-05-15T12:37:52.324783+0000","last_undegraded":"2024-05-15T12:37:52.324783+0000","last_fullsized":"2024-05-15T12:37:52.324783+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:35:55.640503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320225+0000","last_change":"2024-05-15T12:37:11.078505+0000","last_active":"2024-05-15T12:37:52.320225+0000","last_peered":"2024-05-15T12:37:52.320225+0000","last_clean":"2024-05-15T12:37:52.320225+0000","last_became_active":"2024-05-15T12:37:11.078048+0000","last_became_peered":"2024-05-15T12:37:11.078048+0000","last_unstale":"2024-05-15T12:37:52.320225+0000","last_undegraded":"2024-05-15T12:37:52.320225+0000","last_fullsized":"2024-05-15T12:37:52.320225+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-17T00:12:02.632729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544606+0000","last_change":"2024-05-15T12:37:13.097882+0000","last_active":"2024-05-15T12:37:52.544606+0000","last_peered":"2024-05-15T12:37:52.544606+0000","last_clean":"2024-05-15T12:37:52.544606+0000","last_became_active":"2024-05-15T12:37:13.097612+0000","last_became_peered":"2024-05-15T12:37:13.097612+0000","last_unstale":"2024-05-15T12:37:52.544606+0000","last_undegraded":"2024-05-15T12:37:52.544606+0000","last_fullsized":"2024-05-15T12:37:52.544606+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T22:34:37.770323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.319498+0000","last_change":"2024-05-15T12:37:15.100983+0000","last_active":"2024-05-15T12:37:52.319498+0000","last_peered":"2024-05-15T12:37:52.319498+0000","last_clean":"2024-05-15T12:37:52.319498+0000","last_became_active":"2024-05-15T12:37:15.100662+0000","last_became_peered":"2024-05-15T12:37:15.100662+0000","last_unstale":"2024-05-15T12:37:52.319498+0000","last_undegraded":"2024-05-15T12:37:52.319498+0000","last_fullsized":"2024-05-15T12:37:52.319498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:14:28.954709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421142+0000","last_change":"2024-05-15T12:37:09.067662+0000","last_active":"2024-05-15T12:37:52.421142+0000","last_peered":"2024-05-15T12:37:52.421142+0000","last_clean":"2024-05-15T12:37:52.421142+0000","last_became_active":"2024-05-15T12:37:09.067101+0000","last_became_peered":"2024-05-15T12:37:09.067101+0000","last_unstale":"2024-05-15T12:37:52.421142+0000","last_undegraded":"2024-05-15T12:37:52.421142+0000","last_fullsized":"2024-05-15T12:37:52.421142+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:21:22.349984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.428547+0000","last_change":"2024-05-15T12:37:24.674107+0000","last_active":"2024-05-15T12:37:52.428547+0000","last_peered":"2024-05-15T12:37:52.428547+0000","last_clean":"2024-05-15T12:37:52.428547+0000","last_became_active":"2024-05-15T12:37:24.666449+0000","last_became_peered":"2024-05-15T12:37:24.666449+0000","last_unstale":"2024-05-15T12:37:52.428547+0000","last_undegraded":"2024-05-15T12:37:52.428547+0000","last_fullsized":"2024-05-15T12:37:52.428547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:52:59.925765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"68'11","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130533+0000","last_change":"2024-05-15T12:37:13.096029+0000","last_active":"2024-05-15T12:38:34.130533+0000","last_peered":"2024-05-15T12:38:34.130533+0000","last_clean":"2024-05-15T12:38:34.130533+0000","last_became_active":"2024-05-15T12:37:13.095733+0000","last_became_peered":"2024-05-15T12:37:13.095733+0000","last_unstale":"2024-05-15T12:38:34.130533+0000","last_undegraded":"2024-05-15T12:38:34.130533+0000","last_fullsized":"2024-05-15T12:38:34.130533+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:36:00.171088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.433984+0000","last_change":"2024-05-15T12:37:15.317846+0000","last_active":"2024-05-15T12:37:52.433984+0000","last_peered":"2024-05-15T12:37:52.433984+0000","last_clean":"2024-05-15T12:37:52.433984+0000","last_became_active":"2024-05-15T12:37:15.317545+0000","last_became_peered":"2024-05-15T12:37:15.317545+0000","last_unstale":"2024-05-15T12:37:52.433984+0000","last_undegraded":"2024-05-15T12:37:52.433984+0000","last_fullsized":"2024-05-15T12:37:52.433984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:36:07.246378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.320791+0000","last_change":"2024-05-15T12:37:09.072381+0000","last_active":"2024-05-15T12:37:52.320791+0000","last_peered":"2024-05-15T12:37:52.320791+0000","last_clean":"2024-05-15T12:37:52.320791+0000","last_became_active":"2024-05-15T12:37:09.072107+0000","last_became_peered":"2024-05-15T12:37:09.072107+0000","last_unstale":"2024-05-15T12:37:52.320791+0000","last_undegraded":"2024-05-15T12:37:52.320791+0000","last_fullsized":"2024-05-15T12:37:52.320791+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:08:17.775714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323191+0000","last_change":"2024-05-15T12:37:11.085680+0000","last_active":"2024-05-15T12:37:52.323191+0000","last_peered":"2024-05-15T12:37:52.323191+0000","last_clean":"2024-05-15T12:37:52.323191+0000","last_became_active":"2024-05-15T12:37:11.085365+0000","last_became_peered":"2024-05-15T12:37:11.085365+0000","last_unstale":"2024-05-15T12:37:52.323191+0000","last_undegraded":"2024-05-15T12:37:52.323191+0000","last_fullsized":"2024-05-15T12:37:52.323191+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:22:04.818789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'11","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130544+0000","last_change":"2024-05-15T12:37:13.091600+0000","last_active":"2024-05-15T12:38:34.130544+0000","last_peered":"2024-05-15T12:38:34.130544+0000","last_clean":"2024-05-15T12:38:34.130544+0000","last_became_active":"2024-05-15T12:37:13.091320+0000","last_became_peered":"2024-05-15T12:37:13.091320+0000","last_unstale":"2024-05-15T12:38:34.130544+0000","last_undegraded":"2024-05-15T12:38:34.130544+0000","last_fullsized":"2024-05-15T12:38:34.130544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:14:01.722227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323196+0000","last_change":"2024-05-15T12:37:15.103175+0000","last_active":"2024-05-15T12:37:52.323196+0000","last_peered":"2024-05-15T12:37:52.323196+0000","last_clean":"2024-05-15T12:37:52.323196+0000","last_became_active":"2024-05-15T12:37:15.102888+0000","last_became_peered":"2024-05-15T12:37:15.102888+0000","last_unstale":"2024-05-15T12:37:52.323196+0000","last_undegraded":"2024-05-15T12:37:52.323196+0000","last_fullsized":"2024-05-15T12:37:52.323196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:06:55.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.545675+0000","last_change":"2024-05-15T12:37:09.073873+0000","last_active":"2024-05-15T12:37:52.545675+0000","last_peered":"2024-05-15T12:37:52.545675+0000","last_clean":"2024-05-15T12:37:52.545675+0000","last_became_active":"2024-05-15T12:37:09.073514+0000","last_became_peered":"2024-05-15T12:37:09.073514+0000","last_unstale":"2024-05-15T12:37:52.545675+0000","last_undegraded":"2024-05-15T12:37:52.545675+0000","last_fullsized":"2024-05-15T12:37:52.545675+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:29:09.850307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544326+0000","last_change":"2024-05-15T12:37:11.081168+0000","last_active":"2024-05-15T12:37:52.544326+0000","last_peered":"2024-05-15T12:37:52.544326+0000","last_clean":"2024-05-15T12:37:52.544326+0000","last_became_active":"2024-05-15T12:37:11.080867+0000","last_became_peered":"2024-05-15T12:37:11.080867+0000","last_unstale":"2024-05-15T12:37:52.544326+0000","last_undegraded":"2024-05-15T12:37:52.544326+0000","last_fullsized":"2024-05-15T12:37:52.544326+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:57:58.948283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.322971+0000","last_change":"2024-05-15T12:37:13.091686+0000","last_active":"2024-05-15T12:37:52.322971+0000","last_peered":"2024-05-15T12:37:52.322971+0000","last_clean":"2024-05-15T12:37:52.322971+0000","last_became_active":"2024-05-15T12:37:13.091091+0000","last_became_peered":"2024-05-15T12:37:13.091091+0000","last_unstale":"2024-05-15T12:37:52.322971+0000","last_undegraded":"2024-05-15T12:37:52.322971+0000","last_fullsized":"2024-05-15T12:37:52.322971+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:11:24.693744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.420896+0000","last_change":"2024-05-15T12:37:15.109231+0000","last_active":"2024-05-15T12:37:52.420896+0000","last_peered":"2024-05-15T12:37:52.420896+0000","last_clean":"2024-05-15T12:37:52.420896+0000","last_became_active":"2024-05-15T12:37:15.108923+0000","last_became_peered":"2024-05-15T12:37:15.108923+0000","last_unstale":"2024-05-15T12:37:52.420896+0000","last_undegraded":"2024-05-15T12:37:52.420896+0000","last_fullsized":"2024-05-15T12:37:52.420896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:28:25.954595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'11","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130594+0000","last_change":"2024-05-15T12:37:13.091283+0000","last_active":"2024-05-15T12:38:34.130594+0000","last_peered":"2024-05-15T12:38:34.130594+0000","last_clean":"2024-05-15T12:38:34.130594+0000","last_became_active":"2024-05-15T12:37:13.091000+0000","last_became_peered":"2024-05-15T12:37:13.091000+0000","last_unstale":"2024-05-15T12:38:34.130594+0000","last_undegraded":"2024-05-15T12:38:34.130594+0000","last_fullsized":"2024-05-15T12:38:34.130594+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:13:24.211092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544263+0000","last_change":"2024-05-15T12:37:09.069252+0000","last_active":"2024-05-15T12:37:52.544263+0000","last_peered":"2024-05-15T12:37:52.544263+0000","last_clean":"2024-05-15T12:37:52.544263+0000","last_became_active":"2024-05-15T12:37:09.068907+0000","last_became_peered":"2024-05-15T12:37:09.068907+0000","last_unstale":"2024-05-15T12:37:52.544263+0000","last_undegraded":"2024-05-15T12:37:52.544263+0000","last_fullsized":"2024-05-15T12:37:52.544263+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:55:58.609247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.325419+0000","last_change":"2024-05-15T12:37:11.084675+0000","last_active":"2024-05-15T12:37:52.325419+0000","last_peered":"2024-05-15T12:37:52.325419+0000","last_clean":"2024-05-15T12:37:52.325419+0000","last_became_active":"2024-05-15T12:37:11.084429+0000","last_became_peered":"2024-05-15T12:37:11.084429+0000","last_unstale":"2024-05-15T12:37:52.325419+0000","last_undegraded":"2024-05-15T12:37:52.325419+0000","last_fullsized":"2024-05-15T12:37:52.325419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:39:38.526026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544974+0000","last_change":"2024-05-15T12:37:15.316778+0000","last_active":"2024-05-15T12:37:52.544974+0000","last_peered":"2024-05-15T12:37:52.544974+0000","last_clean":"2024-05-15T12:37:52.544974+0000","last_became_active":"2024-05-15T12:37:15.316351+0000","last_became_peered":"2024-05-15T12:37:15.316351+0000","last_unstale":"2024-05-15T12:37:52.544974+0000","last_undegraded":"2024-05-15T12:37:52.544974+0000","last_fullsized":"2024-05-15T12:37:52.544974+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:05:09.440865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.421244+0000","last_change":"2024-05-15T12:37:13.094722+0000","last_active":"2024-05-15T12:37:52.421244+0000","last_peered":"2024-05-15T12:37:52.421244+0000","last_clean":"2024-05-15T12:37:52.421244+0000","last_became_active":"2024-05-15T12:37:13.094410+0000","last_became_peered":"2024-05-15T12:37:13.094410+0000","last_unstale":"2024-05-15T12:37:52.421244+0000","last_undegraded":"2024-05-15T12:37:52.421244+0000","last_fullsized":"2024-05-15T12:37:52.421244+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T23:39:29.669553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323076+0000","last_change":"2024-05-15T12:37:09.071534+0000","last_active":"2024-05-15T12:37:52.323076+0000","last_peered":"2024-05-15T12:37:52.323076+0000","last_clean":"2024-05-15T12:37:52.323076+0000","last_became_active":"2024-05-15T12:37:09.071219+0000","last_became_peered":"2024-05-15T12:37:09.071219+0000","last_unstale":"2024-05-15T12:37:52.323076+0000","last_undegraded":"2024-05-15T12:37:52.323076+0000","last_fullsized":"2024-05-15T12:37:52.323076+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:00:16.664754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324097+0000","last_change":"2024-05-15T12:37:11.087249+0000","last_active":"2024-05-15T12:37:52.324097+0000","last_peered":"2024-05-15T12:37:52.324097+0000","last_clean":"2024-05-15T12:37:52.324097+0000","last_became_active":"2024-05-15T12:37:11.086645+0000","last_became_peered":"2024-05-15T12:37:11.086645+0000","last_unstale":"2024-05-15T12:37:52.324097+0000","last_undegraded":"2024-05-15T12:37:52.324097+0000","last_fullsized":"2024-05-15T12:37:52.324097+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T18:19:27.815519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.429361+0000","last_change":"2024-05-15T12:37:15.103029+0000","last_active":"2024-05-15T12:37:52.429361+0000","last_peered":"2024-05-15T12:37:52.429361+0000","last_clean":"2024-05-15T12:37:52.429361+0000","last_became_active":"2024-05-15T12:37:15.102711+0000","last_became_peered":"2024-05-15T12:37:15.102711+0000","last_unstale":"2024-05-15T12:37:52.429361+0000","last_undegraded":"2024-05-15T12:37:52.429361+0000","last_fullsized":"2024-05-15T12:37:52.429361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T14:03:40.305588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.325084+0000","last_change":"2024-05-15T12:37:13.083388+0000","last_active":"2024-05-15T12:37:52.325084+0000","last_peered":"2024-05-15T12:37:52.325084+0000","last_clean":"2024-05-15T12:37:52.325084+0000","last_became_active":"2024-05-15T12:37:13.082933+0000","last_became_peered":"2024-05-15T12:37:13.082933+0000","last_unstale":"2024-05-15T12:37:52.325084+0000","last_undegraded":"2024-05-15T12:37:52.325084+0000","last_fullsized":"2024-05-15T12:37:52.325084+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:15:53.954674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544350+0000","last_change":"2024-05-15T12:37:09.067077+0000","last_active":"2024-05-15T12:37:52.544350+0000","last_peered":"2024-05-15T12:37:52.544350+0000","last_clean":"2024-05-15T12:37:52.544350+0000","last_became_active":"2024-05-15T12:37:09.066797+0000","last_became_peered":"2024-05-15T12:37:09.066797+0000","last_unstale":"2024-05-15T12:37:52.544350+0000","last_undegraded":"2024-05-15T12:37:52.544350+0000","last_fullsized":"2024-05-15T12:37:52.544350+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T17:25:58.827990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323595+0000","last_change":"2024-05-15T12:37:11.087288+0000","last_active":"2024-05-15T12:37:52.323595+0000","last_peered":"2024-05-15T12:37:52.323595+0000","last_clean":"2024-05-15T12:37:52.323595+0000","last_became_active":"2024-05-15T12:37:11.086950+0000","last_became_peered":"2024-05-15T12:37:11.086950+0000","last_unstale":"2024-05-15T12:37:52.323595+0000","last_undegraded":"2024-05-15T12:37:52.323595+0000","last_fullsized":"2024-05-15T12:37:52.323595+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T15:56:00.337217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.324257+0000","last_change":"2024-05-15T12:37:15.102502+0000","last_active":"2024-05-15T12:37:52.324257+0000","last_peered":"2024-05-15T12:37:52.324257+0000","last_clean":"2024-05-15T12:37:52.324257+0000","last_became_active":"2024-05-15T12:37:15.102133+0000","last_became_peered":"2024-05-15T12:37:15.102133+0000","last_unstale":"2024-05-15T12:37:52.324257+0000","last_undegraded":"2024-05-15T12:37:52.324257+0000","last_fullsized":"2024-05-15T12:37:52.324257+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:14.076605+0000","last_clean_scrub_stamp":"2024-05-15T12:37:14.076605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T20:09:39.802079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'11","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:38:34.130537+0000","last_change":"2024-05-15T12:37:13.093833+0000","last_active":"2024-05-15T12:38:34.130537+0000","last_peered":"2024-05-15T12:38:34.130537+0000","last_clean":"2024-05-15T12:38:34.130537+0000","last_became_active":"2024-05-15T12:37:13.093530+0000","last_became_peered":"2024-05-15T12:37:13.093530+0000","last_unstale":"2024-05-15T12:38:34.130537+0000","last_undegraded":"2024-05-15T12:38:34.130537+0000","last_fullsized":"2024-05-15T12:38:34.130537+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:12.063709+0000","last_clean_scrub_stamp":"2024-05-15T12:37:12.063709+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T19:39:41.201865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"60'1","reported_seq":41,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.544770+0000","last_change":"2024-05-15T12:37:09.067314+0000","last_active":"2024-05-15T12:37:52.544770+0000","last_peered":"2024-05-15T12:37:52.544770+0000","last_clean":"2024-05-15T12:37:52.544770+0000","last_became_active":"2024-05-15T12:37:09.066974+0000","last_became_peered":"2024-05-15T12:37:09.066974+0000","last_unstale":"2024-05-15T12:37:52.544770+0000","last_undegraded":"2024-05-15T12:37:52.544770+0000","last_fullsized":"2024-05-15T12:37:52.544770+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:08.039044+0000","last_clean_scrub_stamp":"2024-05-15T12:37:08.039044+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T12:47:05.111867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-05-15T12:37:52.323640+0000","last_change":"2024-05-15T12:37:11.083458+0000","last_active":"2024-05-15T12:37:52.323640+0000","last_peered":"2024-05-15T12:37:52.323640+0000","last_clean":"2024-05-15T12:37:52.323640+0000","last_became_active":"2024-05-15T12:37:11.083080+0000","last_became_peered":"2024-05-15T12:37:11.083080+0000","last_unstale":"2024-05-15T12:37:52.323640+0000","last_undegraded":"2024-05-15T12:37:52.323640+0000","last_fullsized":"2024-05-15T12:37:52.323640+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-15T12:37:10.051912+0000","last_clean_scrub_stamp":"2024-05-15T12:37:10.051912+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-16T13:08:45.744306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":395,"ondisk_log_size":395,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":82,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712680,"statfs":{"total":95990841344,"available":95961784320,"internally_reserved":0,"allocated":1392640,"data_stored":923479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708536,"statfs":{"total":95990841344,"available":95957540864,"internally_reserved":0,"allocated":1372160,"data_stored":922427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713224,"statfs":{"total":95990841344,"available":95962341376,"internally_reserved":0,"allocated":831488,"data_stored":333226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28380,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712676,"statfs":{"total":95990841344,"available":95961780224,"internally_reserved":0,"allocated":1396736,"data_stored":923698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713280,"statfs":{"total":95990841344,"available":95962398720,"internally_reserved":0,"allocated":774144,"data_stored":331886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713236,"statfs":{"total":95990841344,"available":95962353664,"internally_reserved":0,"allocated":819200,"data_stored":333440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713240,"statfs":{"total":95990841344,"available":95962357760,"internally_reserved":0,"allocated":815104,"data_stored":333552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757282,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713268,"statfs":{"total":95990841344,"available":95962386432,"internally_reserved":0,"allocated":786432,"data_stored":332110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-15T12:38:46.562 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-15T12:38:46.562 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-15T12:38:46.562 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-15T12:38:46.562 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-15T12:38:46.803 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:46.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:46 smithi173 ceph-mon[26787]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:38:46.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:46 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:46.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:46 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:46.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:46 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:46.995 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:46 smithi173 ceph-mon[26787]: Deploying daemon node-exporter.b on smithi173 2024-05-15T12:38:46.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:46 smithi173 systemd[1]: Starting Ceph node-exporter.b for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:38:47.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[31347]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:38:47.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:47.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:47.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:47.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[31347]: Deploying daemon node-exporter.b on smithi173 2024-05-15T12:38:47.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[24799]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-15T12:38:47.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:47.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:47.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:47.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:46 smithi102 ceph-mon[24799]: Deploying daemon node-exporter.b on smithi173 2024-05-15T12:38:47.494 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:47 smithi173 bash[51951]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-05-15T12:38:47.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:47 smithi173 ceph-mon[26787]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:47.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:47 smithi173 bash[51951]: Getting image source signatures 2024-05-15T12:38:47.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:47 smithi173 bash[51951]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-05-15T12:38:47.994 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:47 smithi173 bash[51951]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-05-15T12:38:47.995 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:47 smithi173 bash[51951]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-05-15T12:38:48.012 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:47 smithi102 ceph-mon[24799]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:48.012 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:47 smithi102 ceph-mon[31347]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:49.494 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 bash[51951]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-05-15T12:38:49.495 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 bash[51951]: Writing manifest to image destination 2024-05-15T12:38:49.859 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 podman[51951]: 2024-05-15 12:38:49.555662942 +0000 UTC m=+2.583493308 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-05-15T12:38:49.860 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 podman[51951]: 2024-05-15 12:38:49.646082412 +0000 UTC m=+2.673912783 container create 2ce0d05c5eebe5c797a8e1894cb3b0fa2f269d34082478a2dc895b36bf3ed278 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-15T12:38:49.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:49 smithi102 ceph-mon[31347]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:49.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:49 smithi102 ceph-mon[24799]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:49.938 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-05-15T12:38:49.938 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-15T12:38:50.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-mon[26787]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:50.244 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 podman[51951]: 2024-05-15 12:38:49.898155246 +0000 UTC m=+2.925985596 container init 2ce0d05c5eebe5c797a8e1894cb3b0fa2f269d34082478a2dc895b36bf3ed278 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 podman[51951]: 2024-05-15 12:38:49.901294704 +0000 UTC m=+2.929125072 container start 2ce0d05c5eebe5c797a8e1894cb3b0fa2f269d34082478a2dc895b36bf3ed278 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.905Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.905Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.906Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.906Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.906Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=arp 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=bcache 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=bonding 2024-05-15T12:38:50.245 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=btrfs 2024-05-15T12:38:50.246 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=conntrack 2024-05-15T12:38:50.246 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=cpu 2024-05-15T12:38:50.246 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-05-15T12:38:50.246 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=diskstats 2024-05-15T12:38:50.246 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=dmi 2024-05-15T12:38:50.246 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=edac 2024-05-15T12:38:50.246 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=entropy 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=filefd 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=filesystem 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=hwmon 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=infiniband 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=ipvs 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=loadavg 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=mdadm 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=meminfo 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=netclass 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=netdev 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=netstat 2024-05-15T12:38:50.247 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=nfs 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=nfsd 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=nvme 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=os 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=pressure 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=rapl 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=schedstat 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=selinux 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=sockstat 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=softnet 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=stat 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=tapestats 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=textfile 2024-05-15T12:38:50.248 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=time 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=uname 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=vmstat 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=xfs 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=node_exporter.go:117 level=info collector=zfs 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b[52003]: ts=2024-05-15T12:38:49.907Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 bash[51951]: 2ce0d05c5eebe5c797a8e1894cb3b0fa2f269d34082478a2dc895b36bf3ed278 2024-05-15T12:38:50.249 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:38:49 smithi173 systemd[1]: Started Ceph node-exporter.b for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:38:50.689 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-15T12:38:50.689 INFO:tasks.cephadm:Setup complete, yielding 2024-05-15T12:38:50.690 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-15T12:38:50.703 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi102.front.sepia.ceph.com 2024-05-15T12:38:50.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='client.? 172.21.15.102:0/2109459423' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:38:51.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:38:51.116 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:38:51.116 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:50 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:38:51.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='client.? 172.21.15.102:0/2109459423' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='client.? 172.21.15.102:0/2109459423' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-15T12:38:51.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi102", "caps": []}]: dispatch 2024-05-15T12:38:51.154 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:50 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi173", "caps": []}]: dispatch 2024-05-15T12:38:51.600 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:52.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:52 smithi173 ceph-mon[26787]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:52.128 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:52 smithi173 ceph-mon[26787]: Reconfiguring daemon agent.smithi173 on smithi173 2024-05-15T12:38:52.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:52 smithi173 ceph-mon[26787]: Reconfiguring daemon agent.smithi102 on smithi102 2024-05-15T12:38:52.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:52 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:52.129 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:52 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[31347]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:52.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[31347]: Reconfiguring daemon agent.smithi173 on smithi173 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[31347]: Reconfiguring daemon agent.smithi102 on smithi102 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[24799]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[24799]: Reconfiguring daemon agent.smithi173 on smithi173 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[24799]: Reconfiguring daemon agent.smithi102 on smithi102 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:52.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:52 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:53.541 INFO:teuthology.orchestra.run.smithi102.stdout:167 167 2024-05-15T12:38:53.569 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:53 smithi173 ceph-mon[26787]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:53.569 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:53 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:53.569 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:53 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:53.569 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:38:53 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:38:53.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:53 smithi102 ceph-mon[24799]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:53.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:53 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:53.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:53 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:53.654 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:38:53 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:38:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:38:53.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:53 smithi102 ceph-mon[31347]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:53.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:53 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:38:53.654 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:53 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:38:54.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:54 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:54.533 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'ceph orch status' 2024-05-15T12:38:54.692 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:54 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:54.692 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:54 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:38:55.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:55 smithi102 ceph-mon[31347]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:55.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:55 smithi102 ceph-mon[24799]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:55.738 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:55 smithi173 ceph-mon[26787]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:56.425 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:38:57.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:57 smithi173 ceph-mon[26787]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:57.761 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:57 smithi102 ceph-mon[31347]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:57.761 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:57 smithi102 ceph-mon[24799]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:38:58.968 INFO:teuthology.orchestra.run.smithi102.stdout:Backend: cephadm 2024-05-15T12:38:58.969 INFO:teuthology.orchestra.run.smithi102.stdout:Available: Yes 2024-05-15T12:38:58.969 INFO:teuthology.orchestra.run.smithi102.stdout:Paused: No 2024-05-15T12:38:59.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:59 smithi102 ceph-mon[31347]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:59.652 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:38:59 smithi102 ceph-mon[31347]: from='client.24656 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:38:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:59 smithi102 ceph-mon[24799]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:59.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:38:59 smithi102 ceph-mon[24799]: from='client.24656 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:38:59.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:59 smithi173 ceph-mon[26787]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:38:59.744 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:38:59 smithi173 ceph-mon[26787]: from='client.24656 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:38:59.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-15T12:39:01.267 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:02.402 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:02 smithi102 ceph-mon[31347]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:02.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:02 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:02.403 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:02 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:02.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:02 smithi102 ceph-mon[24799]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:02.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:02 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:02.403 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:02 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:02.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:02 smithi173 ceph-mon[26787]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:02.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:02 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:02.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:02 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:03.674 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:39:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:39:03.743 INFO:teuthology.orchestra.run.smithi102.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-15T12:39:03.743 INFO:teuthology.orchestra.run.smithi102.stdout:agent.smithi102 smithi102 running 2s ago 6m - - 2024-05-15T12:39:03.743 INFO:teuthology.orchestra.run.smithi102.stdout:agent.smithi173 smithi173 starting - - - - 2024-05-15T12:39:03.743 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager.a smithi102 *:9093,9094 running (40s) 2s ago 38s 22.7M - 0.25.0 c8568f914cd2 ed04e8e4535f 2024-05-15T12:39:03.743 INFO:teuthology.orchestra.run.smithi102.stdout:grafana.a smithi173 *:3000 starting - - - - 2024-05-15T12:39:03.743 INFO:teuthology.orchestra.run.smithi102.stdout:iscsi.iscsi.a smithi173 *:5000 running (103s) 93s ago 102s - - 3.8 bce43dd9c04a 0365b95a9400 2024-05-15T12:39:03.743 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.x smithi173 *:8443,8765 running (5m) 93s ago 5m - - 19.0.0-1882-gae248125 bce43dd9c04a c62d9f63c103 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:mgr.y smithi102 *:9283,8765 running (7m) 2s ago 7m 494M - 19.0.0-1882-gae248125 bce43dd9c04a f859a519a094 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:mon.a smithi102 running (7m) 2s ago 7m 48.0M 2048M 19.0.0-1882-gae248125 bce43dd9c04a 6969c783f08f 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:mon.b smithi173 running (6m) 93s ago 6m - 2048M 19.0.0-1882-gae248125 bce43dd9c04a 22e53ca710f1 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:mon.c smithi102 running (5m) 2s ago 5m 43.5M 2048M 19.0.0-1882-gae248125 bce43dd9c04a fa42c120c741 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.a smithi102 *:9100 running (18s) 2s ago 17s 3628k - 1.5.0 0da6a335fe13 3ec530770871 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter.b smithi173 *:9100 starting - - - - 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.0 smithi102 running (5m) 2s ago 5m 49.9M 4096M 19.0.0-1882-gae248125 bce43dd9c04a ae8ce37e1654 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.1 smithi102 running (4m) 2s ago 4m 46.5M 4096M 19.0.0-1882-gae248125 bce43dd9c04a 1b183fe82f24 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.2 smithi102 running (4m) 2s ago 4m 43.2M 4096M 19.0.0-1882-gae248125 bce43dd9c04a f0b02bd35c66 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.3 smithi102 running (3m) 2s ago 3m 42.4M 4096M 19.0.0-1882-gae248125 bce43dd9c04a 6ad9801483fc 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.4 smithi173 running (3m) 93s ago 3m - 4096M 19.0.0-1882-gae248125 bce43dd9c04a f64b8c8ba4cb 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.5 smithi173 running (3m) 93s ago 3m - 4096M 19.0.0-1882-gae248125 bce43dd9c04a 9bf729c63864 2024-05-15T12:39:03.744 INFO:teuthology.orchestra.run.smithi102.stdout:osd.6 smithi173 running (2m) 93s ago 2m - 4096M 19.0.0-1882-gae248125 bce43dd9c04a dc28b82cb115 2024-05-15T12:39:03.745 INFO:teuthology.orchestra.run.smithi102.stdout:osd.7 smithi173 running (2m) 93s ago 2m - 4096M 19.0.0-1882-gae248125 bce43dd9c04a f46ffc322b3f 2024-05-15T12:39:03.745 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus.a smithi173 *:9095 starting - - - - 2024-05-15T12:39:03.745 INFO:teuthology.orchestra.run.smithi102.stdout:rgw.foo.a smithi102 *:80 running (116s) 2s ago 116s 78.0M - 19.0.0-1882-gae248125 bce43dd9c04a cd31bb0d8a84 2024-05-15T12:39:03.745 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:03 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:39:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:39:03.961 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:03 smithi173 ceph-mon[26787]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:03.961 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:03 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:03.961 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:03 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:04.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:03 smithi102 ceph-mon[31347]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:04.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:03 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:04.010 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:03 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:04.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:03 smithi102 ceph-mon[24799]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:04.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:03 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:04.011 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:03 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:04.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-15T12:39:05.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:04 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:05.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:04 smithi102 ceph-mon[31347]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:05.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:04 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:05.139 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:04 smithi102 ceph-mon[24799]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:05.153 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:04 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:05.154 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:04 smithi173 ceph-mon[26787]: from='client.24662 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:05.809 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:06.056 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:05 smithi102 ceph-mon[24799]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:39:06.056 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:05 smithi102 ceph-mon[31347]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:39:06.074 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:05 smithi173 ceph-mon[26787]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:39:07.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:07 smithi102 ceph-mon[31347]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:07.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:07 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:39:07.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:07 smithi102 ceph-mon[24799]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:07.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:07 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:agent 1/2 - 75s * 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:alertmanager ?:9093,9094 1/1 6s ago 98s smithi102=a;count:1 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:grafana ?:3000 0/1 - 94s smithi173=a;count:1 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:iscsi.datapool ?:5000 1/1 97s ago 112s smithi173=iscsi.a;count:1 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:mgr 2/2 97s ago 5m smithi102=y;smithi173=x;count:2 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:mon 3/3 97s ago 6m smithi102:172.21.15.102=a;smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c;smithi173:172.21.15.173=b;count:3 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:node-exporter ?:9100 1/2 - 103s smithi102=a;smithi173=b;count:2 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:osd 8 97s ago - 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:prometheus ?:9095 0/1 - 107s smithi173=a;count:1 2024-05-15T12:39:08.029 INFO:teuthology.orchestra.run.smithi102.stdout:rgw.foo.a ?:80 1/1 6s ago 2m smithi102=foo.a;count:1 2024-05-15T12:39:08.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:07 smithi173 ceph-mon[26787]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:08.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:07 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:39:08.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-15T12:39:08.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:08 smithi102 ceph-mon[31347]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:08.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:08 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:08.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:08 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:08.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:08 smithi102 ceph-mon[24799]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:08.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:08 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:08.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:08 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:09.097 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:08 smithi173 ceph-mon[26787]: from='client.24668 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:09.097 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:08 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:09.097 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:08 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:09.529 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:09.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:09 smithi173 ceph-mon[26787]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:39:09.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:09 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:09.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:09 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:09.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:09 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:09.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:09 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[31347]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[24799]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.123 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:09 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:10.974 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:10 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi102 2024-05-15T12:39:10.974 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:10 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi102 2024-05-15T12:39:10.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:10 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi102 2024-05-15T12:39:11.400 INFO:teuthology.orchestra.run.smithi102.stdout:HOST ADDR LABELS STATUS 2024-05-15T12:39:11.400 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 172.21.15.102 2024-05-15T12:39:11.400 INFO:teuthology.orchestra.run.smithi102.stdout:smithi173 172.21.15.173 2024-05-15T12:39:11.400 INFO:teuthology.orchestra.run.smithi102.stdout:2 hosts in cluster 2024-05-15T12:39:11.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:11 smithi173 ceph-mon[26787]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:12.074 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:11 smithi102 ceph-mon[24799]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:12.075 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:11 smithi102 ceph-mon[31347]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:12.104 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-15T12:39:12.346 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:13.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:12 smithi102 ceph-mon[31347]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:13.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:12 smithi102 ceph-mon[24799]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:13.195 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:12 smithi173 ceph-mon[26787]: from='client.24674 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:13.494 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:39:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:39:13.807 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:13 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:39:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:39:13.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:13 smithi173 ceph-mon[26787]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:13.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:13 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:13.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:13 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:13.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:13 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:39:13.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:13 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:39:13.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:13 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:14.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[31347]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:14.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:14.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[24799]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-15T12:39:14.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:13 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:14.834 INFO:teuthology.orchestra.run.smithi102.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-15T12:39:14.834 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VK400BGN 372G No 5s ago Has a FileSystem, LVM detected 2024-05-15T12:39:14.834 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme1n1 ssd Linux_a873206239e436837536 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:14.834 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme2n1 ssd Linux_a25c89325b5ec4b0cb0d 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:14.834 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme3n1 ssd Linux_113ad7a434a903ab7acf 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:14.834 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102 /dev/nvme4n1 ssd Linux_fc045c8e8a02760fd1ef 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:14.834 INFO:teuthology.orchestra.run.smithi102.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 1s ago Has a FileSystem, LVM detected 2024-05-15T12:39:14.835 INFO:teuthology.orchestra.run.smithi102.stdout:smithi173 /dev/nvme1n1 ssd Linux_a8deeb0d7a44f3b7496c 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:14.835 INFO:teuthology.orchestra.run.smithi102.stdout:smithi173 /dev/nvme2n1 ssd Linux_606caf7cf3cc80abb52c 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:14.835 INFO:teuthology.orchestra.run.smithi102.stdout:smithi173 /dev/nvme3n1 ssd Linux_006111c4528f6f135821 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:14.835 INFO:teuthology.orchestra.run.smithi102.stdout:smithi173 /dev/nvme4n1 ssd Linux_db7ff066689ac85b4735 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-05-15T12:39:15.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[31347]: Detected new or changed devices on smithi173 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[31347]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[31347]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[31347]: Reconfiguring daemon alertmanager.a on smithi102 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[31347]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[24799]: Detected new or changed devices on smithi173 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[24799]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[24799]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[24799]: Reconfiguring daemon alertmanager.a on smithi102 2024-05-15T12:39:15.153 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:14 smithi102 ceph-mon[24799]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-05-15T12:39:15.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:14 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:15.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:14 smithi173 ceph-mon[26787]: Detected new or changed devices on smithi173 2024-05-15T12:39:15.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:14 smithi173 ceph-mon[26787]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-15T12:39:15.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:14 smithi173 ceph-mon[26787]: Reconfiguring alertmanager.a (dependencies changed)... 2024-05-15T12:39:15.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:14 smithi173 ceph-mon[26787]: Reconfiguring daemon alertmanager.a on smithi102 2024-05-15T12:39:15.245 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:14 smithi173 ceph-mon[26787]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-05-15T12:39:15.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls --format yaml' 2024-05-15T12:39:15.814 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:15 smithi102 ceph-mon[24799]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:16.096 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:15 smithi102 ceph-mon[31347]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:16.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:15 smithi173 ceph-mon[26787]: from='client.14811 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:16.531 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:17.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:16 smithi102 ceph-mon[31347]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:17.141 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:16 smithi102 ceph-mon[24799]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:17.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:16 smithi173 ceph-mon[26787]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:19.121 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:18 smithi102 ceph-mon[31347]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:19.122 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:19 smithi102 systemd[1]: Stopping Ceph alertmanager.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:19.122 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:18 smithi102 ceph-mon[24799]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:19.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:18 smithi173 ceph-mon[26787]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: agent 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: agent 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout: host_pattern: '*' 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:37:52.369943Z' 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:01.193611Z' 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout: running: 0 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout: size: 2 2024-05-15T12:39:20.129 INFO:teuthology.orchestra.run.smithi102.stdout:events: 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout:- 2024-05-15T12:37:53.777088Z service:agent [INFO] "service was created" 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: alertmanager 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: alertmanager 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=a 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:37:29.731309Z' 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:01.194163Z' 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-05-15T12:39:20.130 INFO:teuthology.orchestra.run.smithi102.stdout: - 9093 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout: - 9094 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:events: 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:- 2024-05-15T12:38:25.710535Z service:alertmanager [INFO] "service was created" 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: grafana 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: grafana 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi173=a 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout:spec: 2024-05-15T12:39:20.131 INFO:teuthology.orchestra.run.smithi102.stdout: anonymous_access: true 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout: protocol: https 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:37:33.050092Z' 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:08.296323Z' 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout: - 3000 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:events: 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:- 2024-05-15T12:38:40.432612Z service:grafana [INFO] "service was created" 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: iscsi 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:service_id: datapool 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: iscsi.datapool 2024-05-15T12:39:20.132 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi173=iscsi.a 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout:spec: 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: api_password: admin 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: api_port: 5000 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: api_user: admin 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: pool: datapool 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: trusted_ip_list: 172.21.15.173 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:37:15.807723Z' 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:08.296178Z' 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: - 5000 2024-05-15T12:39:20.133 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: mgr 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: mgr 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: count: 2 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=y 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi173=x 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:33:21.776777Z' 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:01.193557Z' 2024-05-15T12:39:20.134 INFO:teuthology.orchestra.run.smithi102.stdout: running: 2 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: size: 2 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: mon 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: mon 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: count: 3 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102:172.21.15.102=a 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102:[v2:172.21.15.102:3301,v1:172.21.15.102:6790]=c 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi173:172.21.15.173=b 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:32:53.710183Z' 2024-05-15T12:39:20.135 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:01.193458Z' 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: running: 3 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: size: 3 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: node-exporter 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: node-exporter 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: count: 2 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=a 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi173=b 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:37:24.360747Z' 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:01.194233Z' 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: - 9100 2024-05-15T12:39:20.136 INFO:teuthology.orchestra.run.smithi102.stdout: running: 2 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: size: 2 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:events: 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:- 2024-05-15T12:38:50.065682Z service:node-exporter [INFO] "service was created" 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: osd 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: osd 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:unmanaged: true 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:spec: 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: filter_logic: AND 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: objectstore: bluestore 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: container_image_id: bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:39:20.137 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:01.193760Z' 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: running: 8 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: size: 8 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: prometheus 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: prometheus 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi173=a 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:37:20.549253Z' 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:08.296252Z' 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-05-15T12:39:20.138 INFO:teuthology.orchestra.run.smithi102.stdout: - 9095 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:events: 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:- 2024-05-15T12:38:50.075503Z service:prometheus [INFO] "service was created" 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:--- 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:service_type: rgw 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:service_id: foo.a 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:service_name: rgw.foo.a 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:placement: 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: count: 1 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: hosts: 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: - smithi102=foo.a 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout:status: 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: created: '2024-05-15T12:37:07.583928Z' 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: last_refresh: '2024-05-15T12:39:01.194094Z' 2024-05-15T12:39:20.139 INFO:teuthology.orchestra.run.smithi102.stdout: ports: 2024-05-15T12:39:20.140 INFO:teuthology.orchestra.run.smithi102.stdout: - 80 2024-05-15T12:39:20.140 INFO:teuthology.orchestra.run.smithi102.stdout: running: 1 2024-05-15T12:39:20.140 INFO:teuthology.orchestra.run.smithi102.stdout: size: 1 2024-05-15T12:39:20.402 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:20 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[59240]: ts=2024-05-15T12:39:20.014Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-05-15T12:39:20.402 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:20 smithi102 podman[69777]: 2024-05-15 12:39:20.044879623 +0000 UTC m=+0.317358033 container died ed04e8e4535f148a21fb99e2bf46896ecf5ca787fa2b4ff855e6daa8da24e9b4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:20.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:20 smithi102 ceph-mon[31347]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:20.902 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:20 smithi102 ceph-mon[31347]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-05-15T12:39:20.903 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:20 smithi102 podman[69777]: 2024-05-15 12:39:20.724239507 +0000 UTC m=+0.996717916 container cleanup ed04e8e4535f148a21fb99e2bf46896ecf5ca787fa2b4ff855e6daa8da24e9b4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:20.903 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:20 smithi102 bash[69777]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a 2024-05-15T12:39:20.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:20 smithi102 ceph-mon[24799]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:20.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:20 smithi102 ceph-mon[24799]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-05-15T12:39:20.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:20 smithi173 ceph-mon[26787]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:20.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:20 smithi173 ceph-mon[26787]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-05-15T12:39:21.402 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:20 smithi102 podman[69811]: 2024-05-15 12:39:20.959050947 +0000 UTC m=+0.915893724 container remove ed04e8e4535f148a21fb99e2bf46896ecf5ca787fa2b4ff855e6daa8da24e9b4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:21.402 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:21 smithi102 podman[69811]: 2024-05-15 12:39:21.059595229 +0000 UTC m=+1.016438007 volume remove bb813f34a21f00086434bac44bc2df4a1c2dd5c4d275a90a436059622c448824 2024-05-15T12:39:21.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-05-15T12:39:21.871 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/mon.a/config 2024-05-15T12:39:21.902 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:21 smithi102 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-15T12:39:21.903 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:21 smithi102 systemd[1]: Stopped Ceph alertmanager.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:39:21.903 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:21 smithi102 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@alertmanager.a.service: Consumed 1.009s CPU time. 2024-05-15T12:39:21.903 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:21 smithi102 systemd[1]: Starting Ceph alertmanager.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:22.220 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:21 smithi102 podman[69967]: 2024-05-15 12:39:21.871276178 +0000 UTC m=+0.021525808 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-05-15T12:39:22.221 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:21 smithi102 podman[69967]: 2024-05-15 12:39:21.989467643 +0000 UTC m=+0.139717298 volume create db10c4a6ae84640f7081dbe48431244f0df1d001f0d1e1c91c68424cc359902c 2024-05-15T12:39:22.221 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 podman[69967]: 2024-05-15 12:39:22.081419207 +0000 UTC m=+0.231668812 container create 6c37c0bb2b5ec344861ecd34baccd5f69c1c5c54e67d5f7b68ac86a41874ad3a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:22.617 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 podman[69967]: 2024-05-15 12:39:22.528212064 +0000 UTC m=+0.678461677 container init 6c37c0bb2b5ec344861ecd34baccd5f69c1c5c54e67d5f7b68ac86a41874ad3a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:22.617 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 podman[69967]: 2024-05-15 12:39:22.531360432 +0000 UTC m=+0.681610045 container start 6c37c0bb2b5ec344861ecd34baccd5f69c1c5c54e67d5f7b68ac86a41874ad3a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:22.617 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.542Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.542Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.544Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.102 port=9094 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.545Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.569Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.569Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.572Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:22.572Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-05-15T12:39:22.618 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 bash[69967]: 6c37c0bb2b5ec344861ecd34baccd5f69c1c5c54e67d5f7b68ac86a41874ad3a 2024-05-15T12:39:22.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[24799]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:22.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:22.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[24799]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:39:22.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:22.903 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[24799]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[31347]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:22.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[31347]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:39:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:22.904 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:22 smithi102 ceph-mon[31347]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:22.904 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:22 smithi102 systemd[1]: Started Ceph alertmanager.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:39:23.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:22 smithi173 ceph-mon[26787]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-05-15T12:39:23.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:22 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:23.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:22 smithi173 ceph-mon[26787]: from='mgr.14625 172.21.15.102:0/1289552911' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-05-15T12:39:23.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:22 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:23.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:22 smithi173 ceph-mon[26787]: from='mgr.14625 ' entity='mgr.y' 2024-05-15T12:39:23.734 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:39:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug there is no tcmu-runner data available 2024-05-15T12:39:23.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:23 smithi102 ceph-mon[31347]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-15T12:39:23.889 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:23 smithi102 ceph-mon[31347]: Reconfiguring daemon prometheus.a on smithi173 2024-05-15T12:39:23.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:23 smithi102 ceph-mon[24799]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-15T12:39:23.889 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:23 smithi102 ceph-mon[24799]: Reconfiguring daemon prometheus.a on smithi173 2024-05-15T12:39:23.889 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:23 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: ::ffff:172.21.15.173 - - [15/May/2024:12:39:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-05-15T12:39:23.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:23 smithi173 ceph-mon[26787]: Reconfiguring prometheus.a (dependencies changed)... 2024-05-15T12:39:23.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:23 smithi173 ceph-mon[26787]: Reconfiguring daemon prometheus.a on smithi173 2024-05-15T12:39:24.760 INFO:teuthology.orchestra.run.smithi102.stdout:osd 8 23s ago - 2024-05-15T12:39:24.761 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:24 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:24.545Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000254928s 2024-05-15T12:39:25.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:24 smithi102 ceph-mon[24799]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:25.073 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:24 smithi102 ceph-mon[24799]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-15T12:39:25.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:24 smithi102 ceph-mon[31347]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:25.073 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:24 smithi102 ceph-mon[31347]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-15T12:39:25.172 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:24 smithi173 ceph-mon[26787]: from='client.24425 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-05-15T12:39:25.172 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:24 smithi173 ceph-mon[26787]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-05-15T12:39:25.380 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-15T12:39:25.391 INFO:tasks.cephadm:Teardown begin 2024-05-15T12:39:25.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:25.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:39:25.451 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-15T12:39:25.451 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T12:39:25.476 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-15T12:39:25.507 INFO:tasks.cephadm:Stopping all daemons... 2024-05-15T12:39:25.508 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-15T12:39:25.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a 2024-05-15T12:39:25.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:25 smithi102 systemd[1]: Stopping Ceph mon.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:26.152 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:25 smithi102 ceph-mon[31347]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:26.152 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:25 smithi102 ceph-mon[24799]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:26.155 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:25 smithi173 ceph-mon[26787]: from='client.24692 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-15T12:39:26.155 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:25 smithi173 systemd[1]: Stopping Ceph prometheus.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:26.472 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.166Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-05-15T12:39:26.473 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[48693]: ts=2024-05-15T12:39:26.167Z caller=main.go:1147 level=info msg="See you next time!" 2024-05-15T12:39:26.474 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 podman[59776]: 2024-05-15 12:39:26.176463619 +0000 UTC m=+0.108529064 container died 8b964ad822b990a674d698c7f544391eb423aa55791cc1e38006220e9821e7bf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:26.652 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:26 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a[24795]: 2024-05-15T12:39:26.244+0000 7f2d014f4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:26.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:26 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a[24795]: 2024-05-15T12:39:26.244+0000 7f2d014f4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-15T12:39:26.653 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:26 smithi102 podman[70297]: 2024-05-15 12:39:26.418569365 +0000 UTC m=+0.272990363 container died 6969c783f08fb4e622124f5fc247feeafcf37b1ef49f3b9347ad68775d785ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:39:26.744 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 podman[59776]: 2024-05-15 12:39:26.582620813 +0000 UTC m=+0.514686252 container cleanup 8b964ad822b990a674d698c7f544391eb423aa55791cc1e38006220e9821e7bf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:26.744 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 bash[59776]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a 2024-05-15T12:39:27.060 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:26 smithi102 podman[70297]: 2024-05-15 12:39:26.803340092 +0000 UTC m=+0.657761090 container cleanup 6969c783f08fb4e622124f5fc247feeafcf37b1ef49f3b9347ad68775d785ae4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=) 2024-05-15T12:39:27.061 INFO:journalctl@ceph.mon.a.smithi102.stdout:May 15 12:39:26 smithi102 bash[70297]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-a 2024-05-15T12:39:27.162 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 podman[59790]: 2024-05-15 12:39:26.808985379 +0000 UTC m=+0.633875283 container remove 8b964ad822b990a674d698c7f544391eb423aa55791cc1e38006220e9821e7bf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:27.163 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-05-15T12:39:27.163 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 systemd[1]: Stopped Ceph prometheus.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:39:27.163 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@prometheus.a.service: Consumed 1.215s CPU time. 2024-05-15T12:39:27.163 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:26 smithi173 systemd[1]: Starting Ceph prometheus.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:27.222 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.a.service' 2024-05-15T12:39:27.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:27.273 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-15T12:39:27.273 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-05-15T12:39:27.273 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.c 2024-05-15T12:39:27.426 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 podman[59879]: 2024-05-15 12:39:27.077383518 +0000 UTC m=+0.018498783 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-05-15T12:39:27.426 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 podman[59879]: 2024-05-15 12:39:27.210272632 +0000 UTC m=+0.151387894 container create 36f44fe93bc7afda148206c721d4dac9b47fd411bfa6d7c034e7137d31446267 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:27.427 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 podman[59879]: 2024-05-15 12:39:27.437542499 +0000 UTC m=+0.378657760 container init 36f44fe93bc7afda148206c721d4dac9b47fd411bfa6d7c034e7137d31446267 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:27.744 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 podman[59879]: 2024-05-15 12:39:27.440510906 +0000 UTC m=+0.381626169 container start 36f44fe93bc7afda148206c721d4dac9b47fd411bfa6d7c034e7137d31446267 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.463Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.463Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.463Z caller=main.go:570 level=info host_details="(Linux 5.14.0-446.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 3 16:10:18 UTC 2024 x86_64 smithi173 (none))" 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.463Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.463Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.464Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.465Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.466Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.466Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.468Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.468Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.574µs 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.468Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-05-15T12:39:27.745 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.468Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.468Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.468Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.96µs wal_replay_duration=474.362µs wbl_replay_duration=342ns total_replay_duration=510.226µs 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.469Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.469Z caller=main.go:1029 level=info msg="TSDB started" 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.469Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.489Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.336954ms db_storage=667ns remote_storage=1.024µs web_handler=265ns query_engine=529ns scrape=410.314µs scrape_sd=90.089µs notify=14.969µs notify_sd=11.756µs rules=18.42326ms tracing=8.883µs 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.489Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:39:27.489Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 bash[59879]: 36f44fe93bc7afda148206c721d4dac9b47fd411bfa6d7c034e7137d31446267 2024-05-15T12:39:27.746 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:39:27 smithi173 systemd[1]: Started Ceph prometheus.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:27 smithi102 systemd[1]: Stopping Ceph mon.c for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-c[31341]: 2024-05-15T12:39:27.574+0000 7ff48e80d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-c[31341]: 2024-05-15T12:39:27.574+0000 7ff48e80d700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:27 smithi102 podman[70418]: 2024-05-15 12:39:27.59545858 +0000 UTC m=+0.131684431 container died fa42c120c7419b9a5e32da87485fd49a0404f555dc890d3e7641a8d917d15958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-c, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_CLEAN=True, ceph=True) 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:39:27] ENGINE Bus STOPPING 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:39:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:39:27] ENGINE Bus STOPPED 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:39:27] ENGINE Bus STARTING 2024-05-15T12:39:27.903 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:39:27] ENGINE Serving on http://:::9283 2024-05-15T12:39:27.904 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:27 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: [15/May/2024:12:39:27] ENGINE Bus STARTED 2024-05-15T12:39:28.217 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:28 smithi102 podman[70418]: 2024-05-15 12:39:28.002037695 +0000 UTC m=+0.538263583 container cleanup fa42c120c7419b9a5e32da87485fd49a0404f555dc890d3e7641a8d917d15958 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-osd-ec-partial-reads-ae24812) 2024-05-15T12:39:28.217 INFO:journalctl@ceph.mon.c.smithi102.stdout:May 15 12:39:28 smithi102 bash[70418]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-c 2024-05-15T12:39:28.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.c.service' 2024-05-15T12:39:28.399 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:28.399 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-05-15T12:39:28.400 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-05-15T12:39:28.400 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.b 2024-05-15T12:39:28.704 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:28 smithi173 systemd[1]: Stopping Ceph mon.b for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:28.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-b[26769]: 2024-05-15T12:39:28.714+0000 7f5f5bf0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:28.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-b[26769]: 2024-05-15T12:39:28.714+0000 7f5f5bf0d700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-05-15T12:39:28.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:28 smithi173 podman[59959]: 2024-05-15 12:39:28.899190133 +0000 UTC m=+0.289790329 container died 22e53ca710f1827f274d866088501c6056715a41b030ee622e922c0cdf2f735a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240514) 2024-05-15T12:39:29.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:29 smithi173 podman[59959]: 2024-05-15 12:39:29.398975949 +0000 UTC m=+0.789576127 container cleanup 22e53ca710f1827f274d866088501c6056715a41b030ee622e922c0cdf2f735a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True) 2024-05-15T12:39:29.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:May 15 12:39:29 smithi173 bash[59959]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mon-b 2024-05-15T12:39:29.780 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mon.b.service' 2024-05-15T12:39:29.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:29.830 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-05-15T12:39:29.830 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-15T12:39:29.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.y 2024-05-15T12:39:30.132 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:29 smithi102 systemd[1]: Stopping Ceph mgr.y for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:30.402 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:30 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:39:30.142+0000 7fa44bf5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:30.403 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:30 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y[25041]: 2024-05-15T12:39:30.142+0000 7fa44bf5b700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-15T12:39:30.403 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:30 smithi102 podman[70813]: 2024-05-15 12:39:30.174510276 +0000 UTC m=+0.126837333 container died f859a519a0948da386644088f4cdbe3958c6798594eff6c87a1c8fe9eea712f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-15T12:39:30.892 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:30 smithi102 podman[70813]: 2024-05-15 12:39:30.593582799 +0000 UTC m=+0.545909840 container cleanup f859a519a0948da386644088f4cdbe3958c6798594eff6c87a1c8fe9eea712f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ae2481253f9b2fa3451954e615eedb67ebdd5314, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:39:30.893 INFO:journalctl@ceph.mgr.y.smithi102.stdout:May 15 12:39:30 smithi102 bash[70813]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-y 2024-05-15T12:39:31.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-15T12:39:31.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:31.069 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-15T12:39:31.069 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-15T12:39:31.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.x 2024-05-15T12:39:31.441 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:39:31 smithi173 systemd[1]: Stopping Ceph mgr.x for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:31.710 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:39:31 smithi173 podman[60105]: 2024-05-15 12:39:31.451801277 +0000 UTC m=+0.171688097 container died c62d9f63c103a63f8574af71d0d5309f2a060acd452a5389373e840823022b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-15T12:39:31.994 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:39:31 smithi173 podman[60105]: 2024-05-15 12:39:31.870470658 +0000 UTC m=+0.590357476 container cleanup c62d9f63c103a63f8574af71d0d5309f2a060acd452a5389373e840823022b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-15T12:39:31.994 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:39:31 smithi173 bash[60105]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x 2024-05-15T12:39:32.268 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:39:32 smithi173 podman[60117]: 2024-05-15 12:39:32.072047665 +0000 UTC m=+0.620044088 container remove c62d9f63c103a63f8574af71d0d5309f2a060acd452a5389373e840823022b3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-mgr-x, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812) 2024-05-15T12:39:32.268 INFO:journalctl@ceph.mgr.x.smithi173.stdout:May 15 12:39:32 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-05-15T12:39:32.288 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-15T12:39:32.324 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:32.324 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-15T12:39:32.324 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-15T12:39:32.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.0 2024-05-15T12:39:32.652 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:39:32 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:39:32.547Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001456883s 2024-05-15T12:39:32.653 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:32 smithi102 systemd[1]: Stopping Ceph osd.0 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:32.653 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:32 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0[35771]: 2024-05-15T12:39:32.604+0000 7f7be6003700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:32.653 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:32 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0[35771]: 2024-05-15T12:39:32.604+0000 7f7be6003700 -1 osd.0 71 *** Got signal Terminated *** 2024-05-15T12:39:32.653 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:32 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0[35771]: 2024-05-15T12:39:32.604+0000 7f7be6003700 -1 osd.0 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:39:38.486 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:38 smithi102 podman[71186]: 2024-05-15 12:39:38.240408017 +0000 UTC m=+5.718106491 container died ae8ce37e165428fb920d7a7b4b0288704370ffa2d2201a619644c3a2714980d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-15T12:39:38.778 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:38 smithi102 podman[71186]: 2024-05-15 12:39:38.587391567 +0000 UTC m=+6.065090041 container cleanup ae8ce37e165428fb920d7a7b4b0288704370ffa2d2201a619644c3a2714980d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-05-15T12:39:38.778 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:38 smithi102 bash[71186]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0 2024-05-15T12:39:39.073 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:38 smithi102 podman[71249]: 2024-05-15 12:39:38.788998493 +0000 UTC m=+0.725140440 container remove ae8ce37e165428fb920d7a7b4b0288704370ffa2d2201a619644c3a2714980d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0, org.label-schema.build-date=20240514, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-15T12:39:39.810 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:39 smithi102 podman[71373]: 2024-05-15 12:39:39.416585856 +0000 UTC m=+0.017857949 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:39:39.810 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:39 smithi102 podman[71373]: 2024-05-15 12:39:39.543568378 +0000 UTC m=+0.144840469 container create a309af3e7fa6cb229b56b9f667f48f3af1037a1c688709d5bed69ce0d0b520a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-15T12:39:39.810 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:39 smithi102 podman[71373]: 2024-05-15 12:39:39.745829874 +0000 UTC m=+0.347101976 container init a309af3e7fa6cb229b56b9f667f48f3af1037a1c688709d5bed69ce0d0b520a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5) 2024-05-15T12:39:39.810 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:39 smithi102 podman[71373]: 2024-05-15 12:39:39.749215739 +0000 UTC m=+0.350487834 container start a309af3e7fa6cb229b56b9f667f48f3af1037a1c688709d5bed69ce0d0b520a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-15T12:39:39.810 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:39 smithi102 podman[71373]: 2024-05-15 12:39:39.820975674 +0000 UTC m=+0.422247773 container attach a309af3e7fa6cb229b56b9f667f48f3af1037a1c688709d5bed69ce0d0b520a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-05-15T12:39:40.902 INFO:journalctl@ceph.osd.0.smithi102.stdout:May 15 12:39:40 smithi102 podman[71373]: 2024-05-15 12:39:40.585749743 +0000 UTC m=+1.187021843 container died a309af3e7fa6cb229b56b9f667f48f3af1037a1c688709d5bed69ce0d0b520a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812) 2024-05-15T12:39:41.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.0.service' 2024-05-15T12:39:41.233 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:41.233 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-15T12:39:41.233 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-15T12:39:41.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.1 2024-05-15T12:39:41.580 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:41 smithi102 systemd[1]: Stopping Ceph osd.1 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:41.902 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1[39163]: 2024-05-15T12:39:41.589+0000 7f699f772700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:41.902 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1[39163]: 2024-05-15T12:39:41.589+0000 7f699f772700 -1 osd.1 71 *** Got signal Terminated *** 2024-05-15T12:39:41.902 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1[39163]: 2024-05-15T12:39:41.589+0000 7f699f772700 -1 osd.1 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:39:47.347 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:47 smithi102 podman[71550]: 2024-05-15 12:39:47.07904951 +0000 UTC m=+5.604061335 container died 1b183fe82f24cb51455468aedda9e8b8e8c2e2cf995187eec54fc2b94c0ad00f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-15T12:39:47.652 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:47 smithi102 podman[71550]: 2024-05-15 12:39:47.557579174 +0000 UTC m=+6.082590989 container cleanup 1b183fe82f24cb51455468aedda9e8b8e8c2e2cf995187eec54fc2b94c0ad00f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:39:47.652 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:47 smithi102 bash[71550]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1 2024-05-15T12:39:48.131 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:47 smithi102 podman[71565]: 2024-05-15 12:39:47.809532921 +0000 UTC m=+0.731689834 container remove 1b183fe82f24cb51455468aedda9e8b8e8c2e2cf995187eec54fc2b94c0ad00f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514) 2024-05-15T12:39:48.403 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:48 smithi102 podman[71748]: 2024-05-15 12:39:48.044910542 +0000 UTC m=+0.018946941 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:39:48.403 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:48 smithi102 podman[71748]: 2024-05-15 12:39:48.143454983 +0000 UTC m=+0.117491372 container create f00c0a01c777582e9eb1ddbf451e70d6a48d34e5ba7bdeead8238073f3e0aaab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-15T12:39:48.403 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:48 smithi102 podman[71748]: 2024-05-15 12:39:48.387848482 +0000 UTC m=+0.361884873 container init f00c0a01c777582e9eb1ddbf451e70d6a48d34e5ba7bdeead8238073f3e0aaab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5) 2024-05-15T12:39:48.403 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:48 smithi102 podman[71748]: 2024-05-15 12:39:48.3911926 +0000 UTC m=+0.365228998 container start f00c0a01c777582e9eb1ddbf451e70d6a48d34e5ba7bdeead8238073f3e0aaab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-15T12:39:48.902 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:48 smithi102 podman[71748]: 2024-05-15 12:39:48.470976694 +0000 UTC m=+0.445013115 container attach f00c0a01c777582e9eb1ddbf451e70d6a48d34e5ba7bdeead8238073f3e0aaab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-15T12:39:48.902 INFO:journalctl@ceph.osd.1.smithi102.stdout:May 15 12:39:48 smithi102 podman[71748]: 2024-05-15 12:39:48.718672882 +0000 UTC m=+0.692709280 container died f00c0a01c777582e9eb1ddbf451e70d6a48d34e5ba7bdeead8238073f3e0aaab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812) 2024-05-15T12:39:49.220 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.1.service' 2024-05-15T12:39:49.261 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:49.261 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-15T12:39:49.261 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-15T12:39:49.261 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.2 2024-05-15T12:39:49.625 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:49 smithi102 systemd[1]: Stopping Ceph osd.2 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:49.902 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:49 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2[42867]: 2024-05-15T12:39:49.635+0000 7f75b3921700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:49.903 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:49 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2[42867]: 2024-05-15T12:39:49.635+0000 7f75b3921700 -1 osd.2 71 *** Got signal Terminated *** 2024-05-15T12:39:49.903 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:49 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2[42867]: 2024-05-15T12:39:49.635+0000 7f75b3921700 -1 osd.2 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:39:55.278 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:55 smithi102 podman[71942]: 2024-05-15 12:39:55.031030496 +0000 UTC m=+5.504972497 container died f0b02bd35c66ff493aee906888e6fe2328d38c2b68a847370793091ef47db0d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812) 2024-05-15T12:39:55.652 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:55 smithi102 podman[71942]: 2024-05-15 12:39:55.488447552 +0000 UTC m=+5.962389542 container cleanup f0b02bd35c66ff493aee906888e6fe2328d38c2b68a847370793091ef47db0d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-15T12:39:55.653 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:55 smithi102 bash[71942]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2 2024-05-15T12:39:56.149 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:55 smithi102 podman[72431]: 2024-05-15 12:39:55.715062331 +0000 UTC m=+0.728721040 container remove f0b02bd35c66ff493aee906888e6fe2328d38c2b68a847370793091ef47db0d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:39:56.845 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:56 smithi102 podman[72601]: 2024-05-15 12:39:56.502593493 +0000 UTC m=+0.016883266 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:39:56.845 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:56 smithi102 podman[72601]: 2024-05-15 12:39:56.611638867 +0000 UTC m=+0.125928625 container create 296b6c9dede4520865483fab486b0783b1bdbbf7db4f7d9d4c413e70b7c8af52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0) 2024-05-15T12:39:56.845 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:56 smithi102 podman[72601]: 2024-05-15 12:39:56.780570241 +0000 UTC m=+0.294860006 container init 296b6c9dede4520865483fab486b0783b1bdbbf7db4f7d9d4c413e70b7c8af52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240514) 2024-05-15T12:39:56.845 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:56 smithi102 podman[72601]: 2024-05-15 12:39:56.784044597 +0000 UTC m=+0.298334355 container start 296b6c9dede4520865483fab486b0783b1bdbbf7db4f7d9d4c413e70b7c8af52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, io.buildah.version=1.33.5) 2024-05-15T12:39:56.845 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:56 smithi102 podman[72601]: 2024-05-15 12:39:56.855594261 +0000 UTC m=+0.369884025 container attach 296b6c9dede4520865483fab486b0783b1bdbbf7db4f7d9d4c413e70b7c8af52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-15T12:39:57.152 INFO:journalctl@ceph.osd.2.smithi102.stdout:May 15 12:39:57 smithi102 podman[72601]: 2024-05-15 12:39:57.077821004 +0000 UTC m=+0.592110771 container died 296b6c9dede4520865483fab486b0783b1bdbbf7db4f7d9d4c413e70b7c8af52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514) 2024-05-15T12:39:57.870 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.2.service' 2024-05-15T12:39:57.912 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:39:57.912 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-15T12:39:57.912 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-15T12:39:57.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.3 2024-05-15T12:39:58.152 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:39:58 smithi102 systemd[1]: Stopping Ceph osd.3 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:39:58.652 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:39:58 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3[46488]: 2024-05-15T12:39:58.245+0000 7f87ccffb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:39:58.652 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:39:58 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3[46488]: 2024-05-15T12:39:58.245+0000 7f87ccffb700 -1 osd.3 71 *** Got signal Terminated *** 2024-05-15T12:39:58.653 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:39:58 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3[46488]: 2024-05-15T12:39:58.245+0000 7f87ccffb700 -1 osd.3 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:39:59.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:39:59 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:39:59.279+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:39:59.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:39:59 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:39:59.330+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:00.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:00 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:00.244+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:00.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:00 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:00.362+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:00.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:00 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:00.354+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:01.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:01 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:01.284+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:01.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:01 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:01.324+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:01.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:01 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:01.335+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:01.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:01 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:01.244+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:01.652 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:01 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3[46488]: 2024-05-15T12:40:01.331+0000 7f87c95ef700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.239214+0000 front 2024-05-15T12:39:35.239040+0000 (oldest deadline 2024-05-15T12:40:01.139249+0000) 2024-05-15T12:40:02.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:02 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:02.325+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:02.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:02 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:02.353+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:02.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:02 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:02.286+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:02.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:02 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:02.199+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:02.652 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:02 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3[46488]: 2024-05-15T12:40:02.300+0000 7f87c95ef700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.239214+0000 front 2024-05-15T12:39:35.239040+0000 (oldest deadline 2024-05-15T12:40:01.139249+0000) 2024-05-15T12:40:03.494 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:03.239+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:03.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:03.311+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:03.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:03.341+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:03.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:03 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:03.262+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:04.029 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:03 smithi102 podman[72705]: 2024-05-15 12:40:03.776225419 +0000 UTC m=+5.628480508 container died 6ad9801483fc76c1d24843b18433064bf53b28211a27a6234109c4695992455d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-15T12:40:04.386 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:04 smithi102 podman[72705]: 2024-05-15 12:40:04.212701978 +0000 UTC m=+6.064957062 container cleanup 6ad9801483fc76c1d24843b18433064bf53b28211a27a6234109c4695992455d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-15T12:40:04.387 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:04 smithi102 bash[72705]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3 2024-05-15T12:40:04.494 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:04 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:04.334+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:04.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:04 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:04.291+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:04.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:04 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:04.230+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:04.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:04 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:04.287+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:04.652 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:04 smithi102 podman[72721]: 2024-05-15 12:40:04.397384256 +0000 UTC m=+0.610363339 container remove 6ad9801483fc76c1d24843b18433064bf53b28211a27a6234109c4695992455d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-15T12:40:05.089 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:04 smithi102 podman[72811]: 2024-05-15 12:40:04.831490747 +0000 UTC m=+0.110465389 container create 43ab7eb6a57aa28502c25080f103f36cb9ec561a44c488b36a4684768bee2c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5) 2024-05-15T12:40:05.090 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:04 smithi102 podman[72811]: 2024-05-15 12:40:04.738316734 +0000 UTC m=+0.017291393 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:40:05.090 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:05 smithi102 podman[72811]: 2024-05-15 12:40:05.025799126 +0000 UTC m=+0.304773769 container init 43ab7eb6a57aa28502c25080f103f36cb9ec561a44c488b36a4684768bee2c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-15T12:40:05.090 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:05 smithi102 podman[72811]: 2024-05-15 12:40:05.029019189 +0000 UTC m=+0.307993841 container start 43ab7eb6a57aa28502c25080f103f36cb9ec561a44c488b36a4684768bee2c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-15T12:40:05.090 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:05 smithi102 podman[72811]: 2024-05-15 12:40:05.100571237 +0000 UTC m=+0.379545899 container attach 43ab7eb6a57aa28502c25080f103f36cb9ec561a44c488b36a4684768bee2c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-osd-ec-partial-reads-ae24812, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2) 2024-05-15T12:40:05.402 INFO:journalctl@ceph.osd.3.smithi102.stdout:May 15 12:40:05 smithi102 podman[72811]: 2024-05-15 12:40:05.326614759 +0000 UTC m=+0.605589409 container died 43ab7eb6a57aa28502c25080f103f36cb9ec561a44c488b36a4684768bee2c85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-3-deactivate, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-15T12:40:05.744 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:05 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:05.262+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:05.745 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:05 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:05.332+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:05.745 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:05 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:05.308+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:05.746 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:05 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:05.270+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:05.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.3.service' 2024-05-15T12:40:05.826 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:40:05.826 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-15T12:40:05.826 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-15T12:40:05.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.4 2024-05-15T12:40:06.136 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:05 smithi173 systemd[1]: Stopping Ceph osd.4 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:06.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:06 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:06.276+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:06.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:06 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:06.312+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:06.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:06 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:06.306+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:06.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:06 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:06.146+0000 7efeff578700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:40:06.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:06 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:06.146+0000 7efeff578700 -1 osd.4 71 *** Got signal Terminated *** 2024-05-15T12:40:06.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:06 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:06.146+0000 7efeff578700 -1 osd.4 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:40:06.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:06 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:06.279+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:07.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:07 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:07.281+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:07.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:07 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:07.238+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:07.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:07 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:07.342+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:07.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:07 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:07.311+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:08.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:08 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:08.276+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:08.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:08 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:08.316+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:08.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:08 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:08.310+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:08.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:08 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:08.224+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:08.495 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:08 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:08.224+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:44.569123+0000 front 2024-05-15T12:39:44.569176+0000 (oldest deadline 2024-05-15T12:40:07.969633+0000) 2024-05-15T12:40:09.494 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:09 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:09.210+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:09.494 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:09 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:09.210+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:44.569123+0000 front 2024-05-15T12:39:44.569176+0000 (oldest deadline 2024-05-15T12:40:07.969633+0000) 2024-05-15T12:40:09.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:09 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:09.230+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:09.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:09 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:09.230+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:09.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:09 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:09.328+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:09.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:09 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:09.328+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:09.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:09 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:09.344+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:10.494 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:10 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:10.210+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:34.667721+0000 front 2024-05-15T12:39:34.667693+0000 (oldest deadline 2024-05-15T12:40:00.567615+0000) 2024-05-15T12:40:10.494 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:10 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4[33545]: 2024-05-15T12:40:10.210+0000 7efefbb6c700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:44.569123+0000 front 2024-05-15T12:39:44.569176+0000 (oldest deadline 2024-05-15T12:40:07.969633+0000) 2024-05-15T12:40:10.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:10 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:10.202+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:10.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:10 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:10.202+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:10.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:10 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:10.284+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:10.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:10 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:10.284+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:10.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:10 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:10.303+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:11.494 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:11 smithi173 podman[61238]: 2024-05-15 12:40:11.466019507 +0000 UTC m=+5.426338885 container died f64b8c8ba4cb9b8263c9af42ea2cd7b826cb8a71ff9d3b0189b36af90aae13a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-osd-ec-partial-reads-ae24812, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True) 2024-05-15T12:40:11.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:11 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:11.290+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:11.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:11 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:11.290+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:11.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:11 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:11.216+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:11.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:11 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:11.216+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:11.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:11 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:11.265+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:11.994 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:11 smithi173 podman[61238]: 2024-05-15 12:40:11.896783562 +0000 UTC m=+5.857102983 container cleanup f64b8c8ba4cb9b8263c9af42ea2cd7b826cb8a71ff9d3b0189b36af90aae13a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-15T12:40:11.994 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:11 smithi173 bash[61238]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4 2024-05-15T12:40:12.494 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:12 smithi173 podman[61255]: 2024-05-15 12:40:12.098094147 +0000 UTC m=+0.631446477 container remove f64b8c8ba4cb9b8263c9af42ea2cd7b826cb8a71ff9d3b0189b36af90aae13a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514) 2024-05-15T12:40:12.494 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:12 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:12.248+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:12.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:12 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:12.236+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:12.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:12 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:12.236+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:12.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:12 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:12.316+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:12.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:12 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:12.316+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:12.808 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:12 smithi173 podman[61343]: 2024-05-15 12:40:12.423270506 +0000 UTC m=+0.017546311 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:40:12.808 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:12 smithi173 podman[61343]: 2024-05-15 12:40:12.524260202 +0000 UTC m=+0.118536000 container create 91e83979274bc49353068f8b27724de70384c7edcea86d3f95414bf75b3ced0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-15T12:40:12.808 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:12 smithi173 podman[61343]: 2024-05-15 12:40:12.743774545 +0000 UTC m=+0.338050344 container init 91e83979274bc49353068f8b27724de70384c7edcea86d3f95414bf75b3ced0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-15T12:40:12.808 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:12 smithi173 podman[61343]: 2024-05-15 12:40:12.746984435 +0000 UTC m=+0.341260234 container start 91e83979274bc49353068f8b27724de70384c7edcea86d3f95414bf75b3ced0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux ) 2024-05-15T12:40:12.808 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:12 smithi173 podman[61343]: 2024-05-15 12:40:12.818523045 +0000 UTC m=+0.412798858 container attach 91e83979274bc49353068f8b27724de70384c7edcea86d3f95414bf75b3ced0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-15T12:40:13.231 INFO:journalctl@ceph.osd.4.smithi173.stdout:May 15 12:40:13 smithi173 podman[61343]: 2024-05-15 12:40:13.050306688 +0000 UTC m=+0.644582486 container died 91e83979274bc49353068f8b27724de70384c7edcea86d3f95414bf75b3ced0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-15T12:40:13.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:13.244+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:13.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:13.244+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:13.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:13.348+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:13.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:13.348+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:13.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:13.241+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:13.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:13 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:13.241+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:13.737 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.4.service' 2024-05-15T12:40:13.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:40:13.779 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-15T12:40:13.779 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-15T12:40:13.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.5 2024-05-15T12:40:13.994 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:13 smithi173 systemd[1]: Stopping Ceph osd.5 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:14.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:14.219+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:14.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:14.219+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:14.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:14.278+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:14.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:14.278+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:14.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:14.133+0000 7f220ec67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:40:14.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:14.133+0000 7f220ec67700 -1 osd.5 71 *** Got signal Terminated *** 2024-05-15T12:40:14.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:14.133+0000 7f220ec67700 -1 osd.5 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:40:14.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:14.321+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:14.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:14 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:14.321+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:15.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:15 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:15.335+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:15.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:15 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:15.335+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:15.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:15 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:15.286+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:15.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:15 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:15.286+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:15.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:15 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:15.227+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:15.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:15 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:15.227+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:16.557 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:16 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:16.257+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:16.557 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:16 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:16.257+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:16.558 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:16 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:16.268+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:16.558 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:16 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:16.268+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:16.558 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:16 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:16.368+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:16.558 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:16 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:16.368+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:16.558 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:16 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:16.368+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.152855+0000 front 2024-05-15T12:39:53.152885+0000 (oldest deadline 2024-05-15T12:40:15.453083+0000) 2024-05-15T12:40:17.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:17 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:17.344+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:17.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:17 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:17.344+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:17.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:17 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:17.344+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.152855+0000 front 2024-05-15T12:39:53.152885+0000 (oldest deadline 2024-05-15T12:40:15.453083+0000) 2024-05-15T12:40:17.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:17 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:17.256+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:17.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:17 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:17.256+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:17.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:17 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:17.251+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:17.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:17 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:17.251+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:18.452 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:18 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:18.239+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:18.453 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:18 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:18.239+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:18.453 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:18 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:18.318+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:18.453 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:18 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:18.318+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:18.453 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:18 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:18.318+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.152855+0000 front 2024-05-15T12:39:53.152885+0000 (oldest deadline 2024-05-15T12:40:15.453083+0000) 2024-05-15T12:40:18.453 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:18 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:18.212+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:18.453 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:18 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:18.212+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:19.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:19.197+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:19.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:19.197+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:19.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:19.212+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:19.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:19.212+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:19.498 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: *** Caught signal (Segmentation fault) ** 2024-05-15T12:40:19.498 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: in thread 7f21f6231700 thread_name:safe_timer 2024-05-15T12:40:19.498 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev) 2024-05-15T12:40:19.498 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22160bed20] 2024-05-15T12:40:19.499 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x56053b69e60a] 2024-05-15T12:40:19.499 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x56053b72c498] 2024-05-15T12:40:19.499 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x56053b3100c3] 2024-05-15T12:40:19.499 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x56053b59c151] 2024-05-15T12:40:19.499 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x56053b28ac44] 2024-05-15T12:40:19.499 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x56053b28cc9a] 2024-05-15T12:40:19.499 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x56053b28ec03] 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x56053b1f079a] 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x56053b84c92f] 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 11: (CommonSafeTimerThread::entry()+0x11) [0x56053b84dc51] 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f22160b41ca] 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 13: clone() 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:19.147+0000 7f21f6231700 -1 *** Caught signal (Segmentation fault) ** 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: in thread 7f21f6231700 thread_name:safe_timer 2024-05-15T12:40:19.500 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:19.501 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev) 2024-05-15T12:40:19.501 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22160bed20] 2024-05-15T12:40:19.501 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x56053b69e60a] 2024-05-15T12:40:19.501 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x56053b72c498] 2024-05-15T12:40:19.501 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x56053b3100c3] 2024-05-15T12:40:19.501 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x56053b59c151] 2024-05-15T12:40:19.501 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x56053b28ac44] 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x56053b28cc9a] 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x56053b28ec03] 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x56053b1f079a] 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x56053b84c92f] 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 11: (CommonSafeTimerThread::entry()+0x11) [0x56053b84dc51] 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f22160b41ca] 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 13: clone() 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:19.502 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -3542> 2024-05-15T12:39:59.279+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -3394> 2024-05-15T12:40:00.244+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -3067> 2024-05-15T12:40:01.284+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -2936> 2024-05-15T12:40:02.325+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -2696> 2024-05-15T12:40:03.311+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -2531> 2024-05-15T12:40:04.287+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -2182> 2024-05-15T12:40:05.270+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -2146> 2024-05-15T12:40:06.312+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1902> 2024-05-15T12:40:07.342+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1840> 2024-05-15T12:40:08.316+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1560> 2024-05-15T12:40:09.328+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1559> 2024-05-15T12:40:09.328+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1501> 2024-05-15T12:40:10.284+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1500> 2024-05-15T12:40:10.284+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.503 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1264> 2024-05-15T12:40:11.290+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1263> 2024-05-15T12:40:11.290+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1130> 2024-05-15T12:40:12.316+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -1129> 2024-05-15T12:40:12.316+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -854> 2024-05-15T12:40:13.348+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -853> 2024-05-15T12:40:13.348+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -798> 2024-05-15T12:40:14.133+0000 7f220ec67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -797> 2024-05-15T12:40:14.133+0000 7f220ec67700 -1 osd.5 71 *** Got signal Terminated *** 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -795> 2024-05-15T12:40:14.133+0000 7f220ec67700 -1 osd.5 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -775> 2024-05-15T12:40:14.321+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -774> 2024-05-15T12:40:14.321+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -636> 2024-05-15T12:40:15.335+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -635> 2024-05-15T12:40:15.335+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.504 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -602> 2024-05-15T12:40:16.368+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -601> 2024-05-15T12:40:16.368+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -600> 2024-05-15T12:40:16.368+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.152855+0000 front 2024-05-15T12:39:53.152885+0000 (oldest deadline 2024-05-15T12:40:15.453083+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -285> 2024-05-15T12:40:17.344+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -284> 2024-05-15T12:40:17.344+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -283> 2024-05-15T12:40:17.344+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.152855+0000 front 2024-05-15T12:39:53.152885+0000 (oldest deadline 2024-05-15T12:40:15.453083+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -216> 2024-05-15T12:40:18.318+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:33.950281+0000 front 2024-05-15T12:39:33.950361+0000 (oldest deadline 2024-05-15T12:39:59.250192+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -215> 2024-05-15T12:40:18.318+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.251641+0000 front 2024-05-15T12:39:46.251758+0000 (oldest deadline 2024-05-15T12:40:09.151561+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: -214> 2024-05-15T12:40:18.318+0000 7f2207253700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.152855+0000 front 2024-05-15T12:39:53.152885+0000 (oldest deadline 2024-05-15T12:40:15.453083+0000) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 0> 2024-05-15T12:40:19.147+0000 7f21f6231700 -1 *** Caught signal (Segmentation fault) ** 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: in thread 7f21f6231700 thread_name:safe_timer 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: ceph version 19.0.0-1882-gae248125 (ae2481253f9b2fa3451954e615eedb67ebdd5314) squid (dev) 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f22160bed20] 2024-05-15T12:40:19.505 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x56053b69e60a] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x56053b72c498] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x56053b3100c3] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x56053b59c151] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x56053b28ac44] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x56053b28cc9a] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x56053b28ec03] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x56053b1f079a] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x56053b84c92f] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 11: (CommonSafeTimerThread::entry()+0x11) [0x56053b84dc51] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f22160b41ca] 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 13: clone() 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-05-15T12:40:19.506 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5[37069]: 2024-05-15T12:40:19.507 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 podman[61447]: 2024-05-15 12:40:19.209094487 +0000 UTC m=+5.194625956 container died 9bf729c638642b7667e75faa9d78b5664089e52d73c6452b1680afc011aee703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-osd-ec-partial-reads-ae24812, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-15T12:40:19.905 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 podman[61447]: 2024-05-15 12:40:19.665057259 +0000 UTC m=+5.650588714 container cleanup 9bf729c638642b7667e75faa9d78b5664089e52d73c6452b1680afc011aee703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-15T12:40:19.905 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 bash[61447]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5 2024-05-15T12:40:20.183 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 podman[61683]: 2024-05-15 12:40:19.916469496 +0000 UTC m=+0.698577690 container remove 9bf729c638642b7667e75faa9d78b5664089e52d73c6452b1680afc011aee703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-15T12:40:20.183 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:19 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.5.service: Main process exited, code=exited, status=139/n/a 2024-05-15T12:40:20.494 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:20 smithi173 podman[61830]: 2024-05-15 12:40:20.158904493 +0000 UTC m=+0.017708912 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:40:20.495 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:20 smithi173 podman[61830]: 2024-05-15 12:40:20.284285915 +0000 UTC m=+0.143090334 container create 8d70732d413803179ce42c288bdb23567307be6e415647167d9f20ffe630d9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-15T12:40:20.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:20 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:20.193+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:20.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:20 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:20.193+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:20.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:20 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:20.193+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:54.169158+0000 front 2024-05-15T12:39:54.169079+0000 (oldest deadline 2024-05-15T12:40:19.468922+0000) 2024-05-15T12:40:20.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:20 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:20.212+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:20.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:20 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:20.212+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:20.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:20 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:20.212+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:20.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:20 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:20.212+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:20.827 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:20 smithi173 podman[61830]: 2024-05-15 12:40:20.512083121 +0000 UTC m=+0.370887541 container init 8d70732d413803179ce42c288bdb23567307be6e415647167d9f20ffe630d9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-15T12:40:20.827 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:20 smithi173 podman[61830]: 2024-05-15 12:40:20.516076359 +0000 UTC m=+0.374880779 container start 8d70732d413803179ce42c288bdb23567307be6e415647167d9f20ffe630d9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:40:20.827 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:20 smithi173 podman[61830]: 2024-05-15 12:40:20.628799016 +0000 UTC m=+0.487603446 container attach 8d70732d413803179ce42c288bdb23567307be6e415647167d9f20ffe630d9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:40:21.138 INFO:journalctl@ceph.osd.5.smithi173.stdout:May 15 12:40:20 smithi173 podman[61830]: 2024-05-15 12:40:20.837556472 +0000 UTC m=+0.696360894 container died 8d70732d413803179ce42c288bdb23567307be6e415647167d9f20ffe630d9fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-osd-ec-partial-reads-ae24812, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-05-15T12:40:21.421 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:21.182+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:21.421 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:21.182+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:21.422 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:21.182+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:54.169158+0000 front 2024-05-15T12:39:54.169079+0000 (oldest deadline 2024-05-15T12:40:19.468922+0000) 2024-05-15T12:40:21.422 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:21.202+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:21.422 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:21.202+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:21.422 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:21.202+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:21.422 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:21.202+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:21.448 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.5.service' 2024-05-15T12:40:21.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:40:21.488 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-15T12:40:21.488 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-15T12:40:21.488 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.6 2024-05-15T12:40:21.744 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:21 smithi173 systemd[1]: Stopping Ceph osd.6 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:22.219 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:21.869+0000 7fb798c0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:40:22.220 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:21.869+0000 7fb798c0b700 -1 osd.6 71 *** Got signal Terminated *** 2024-05-15T12:40:22.220 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:21 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:21.869+0000 7fb798c0b700 -1 osd.6 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:40:22.494 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:22.231+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:22.494 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:22.231+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:22.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:22.231+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:54.169158+0000 front 2024-05-15T12:39:54.169079+0000 (oldest deadline 2024-05-15T12:40:19.468922+0000) 2024-05-15T12:40:22.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:22.230+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:22.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:22.230+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:22.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:22.230+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:22.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:22 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:22.230+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:23.477 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:23.186+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:23.478 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:23.186+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:23.478 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:23.186+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:54.169158+0000 front 2024-05-15T12:39:54.169079+0000 (oldest deadline 2024-05-15T12:40:19.468922+0000) 2024-05-15T12:40:23.478 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:23.237+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:23.478 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:23.237+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:23.478 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:23.237+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:23.478 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:23 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:23.237+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:24.301 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:24 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:24.206+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:24.301 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:24 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:24.206+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:24.301 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:24 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:24.206+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:24.301 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:24 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:24.206+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:24.302 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:24 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:24.155+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:24.302 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:24 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:24.155+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:24.302 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:24 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:24.155+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:54.169158+0000 front 2024-05-15T12:39:54.169079+0000 (oldest deadline 2024-05-15T12:40:19.468922+0000) 2024-05-15T12:40:25.455 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:25.148+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:25.455 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:25.148+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:25.455 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:25.148+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:54.169158+0000 front 2024-05-15T12:39:54.169079+0000 (oldest deadline 2024-05-15T12:40:19.468922+0000) 2024-05-15T12:40:25.455 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:25.148+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:39:59.469404+0000 front 2024-05-15T12:39:59.469429+0000 (oldest deadline 2024-05-15T12:40:24.769346+0000) 2024-05-15T12:40:25.455 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:25.215+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:25.455 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:25.215+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:25.455 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:25.215+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:25.456 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:25 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:25.215+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:26.244 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:26.155+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:35.066264+0000 front 2024-05-15T12:39:35.066255+0000 (oldest deadline 2024-05-15T12:39:58.566010+0000) 2024-05-15T12:40:26.244 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:26.155+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:46.568225+0000 front 2024-05-15T12:39:46.568161+0000 (oldest deadline 2024-05-15T12:40:12.468021+0000) 2024-05-15T12:40:26.245 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:26.155+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:54.169158+0000 front 2024-05-15T12:39:54.169079+0000 (oldest deadline 2024-05-15T12:40:19.468922+0000) 2024-05-15T12:40:26.245 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6[40587]: 2024-05-15T12:40:26.155+0000 7fb7911f7700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:39:59.469404+0000 front 2024-05-15T12:39:59.469429+0000 (oldest deadline 2024-05-15T12:40:24.769346+0000) 2024-05-15T12:40:26.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:26.262+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:26.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:26.262+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:26.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:26.262+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:26.745 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:26 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:26.262+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:27.495 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:27 smithi173 podman[62026]: 2024-05-15 12:40:27.403801188 +0000 UTC m=+5.674858829 container died dc28b82cb115265bcda573f166a58b7b2b7d699dee744403ad30caae4b7d3ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-15T12:40:27.495 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:27.473Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-05-15T12:40:27.495 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:27.474Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-05-15T12:40:27.495 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:27.475Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-05-15T12:40:27.495 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:27.477Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-05-15T12:40:27.495 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:27.477Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.102:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.102:8765: connect: connection refused" 2024-05-15T12:40:27.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:27.235+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:27.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:27.235+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:27.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:27.235+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:27.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:27.235+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:27.496 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:27 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:27.235+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:27.994 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:27 smithi173 podman[62026]: 2024-05-15 12:40:27.893818961 +0000 UTC m=+6.164876644 container cleanup dc28b82cb115265bcda573f166a58b7b2b7d699dee744403ad30caae4b7d3ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6, org.label-schema.build-date=20240514, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-15T12:40:27.994 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:27 smithi173 bash[62026]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6 2024-05-15T12:40:28.451 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:28 smithi173 podman[62491]: 2024-05-15 12:40:28.128328606 +0000 UTC m=+0.716372763 container remove dc28b82cb115265bcda573f166a58b7b2b7d699dee744403ad30caae4b7d3ce8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-15T12:40:28.452 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:28.219+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:28.452 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:28.219+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:28.452 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:28.219+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:28.452 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:28.219+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:28.452 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:28 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:28.219+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:28.737 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:28 smithi173 podman[62581]: 2024-05-15 12:40:28.462560875 +0000 UTC m=+0.108325202 container create 53302a4e1ebc4c0c27803c688e17180a2b7edbb9d6ad77519a8af15d31e05b01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-15T12:40:28.737 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:28 smithi173 podman[62581]: 2024-05-15 12:40:28.371415482 +0000 UTC m=+0.017179818 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:40:28.738 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:28 smithi173 podman[62581]: 2024-05-15 12:40:28.673336781 +0000 UTC m=+0.319101110 container init 53302a4e1ebc4c0c27803c688e17180a2b7edbb9d6ad77519a8af15d31e05b01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240514, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-15T12:40:28.738 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:28 smithi173 podman[62581]: 2024-05-15 12:40:28.676760077 +0000 UTC m=+0.322524406 container start 53302a4e1ebc4c0c27803c688e17180a2b7edbb9d6ad77519a8af15d31e05b01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-15T12:40:28.738 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:28 smithi173 podman[62581]: 2024-05-15 12:40:28.748245097 +0000 UTC m=+0.394009434 container attach 53302a4e1ebc4c0c27803c688e17180a2b7edbb9d6ad77519a8af15d31e05b01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-15T12:40:28.994 INFO:journalctl@ceph.osd.6.smithi173.stdout:May 15 12:40:28 smithi173 podman[62581]: 2024-05-15 12:40:28.970407378 +0000 UTC m=+0.616171707 container died 53302a4e1ebc4c0c27803c688e17180a2b7edbb9d6ad77519a8af15d31e05b01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-osd-ec-partial-reads-ae24812, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-15T12:40:29.483 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.189+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:29.483 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.189+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:29.483 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.189+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:29.483 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.189+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:29.483 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.189+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:29.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.6.service' 2024-05-15T12:40:29.549 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:40:29.549 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-15T12:40:29.549 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-15T12:40:29.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.7 2024-05-15T12:40:29.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 systemd[1]: Stopping Ceph osd.7 for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:30.224 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.913+0000 7f5992a76700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:40:30.224 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.913+0000 7f5992a76700 -1 osd.7 71 *** Got signal Terminated *** 2024-05-15T12:40:30.224 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:29 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:29.913+0000 7f5992a76700 -1 osd.7 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-15T12:40:30.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:30.233+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:30.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:30.233+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:30.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:30.233+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:30.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:30.233+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:30.495 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:30 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:30.233+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:31.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:31.277+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:31.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:31.277+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:31.745 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:31.277+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:31.745 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:31.277+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:31.745 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:31 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:31.278+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:32.646 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:32.296+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:32.647 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:32.296+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:32.647 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:32.296+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:32.647 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:32.296+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:32.647 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:32.296+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.173:6804 osd.4 since back 2024-05-15T12:40:06.475566+0000 front 2024-05-15T12:40:06.475500+0000 (oldest deadline 2024-05-15T12:40:31.775599+0000) 2024-05-15T12:40:32.647 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:32 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:32.296+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:33.658 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:33 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:33.304+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:33.658 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:33 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:33.304+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:33.658 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:33 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:33.304+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:33.658 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:33 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:33.304+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:33.658 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:33 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:33.304+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.173:6804 osd.4 since back 2024-05-15T12:40:06.475566+0000 front 2024-05-15T12:40:06.475500+0000 (oldest deadline 2024-05-15T12:40:31.775599+0000) 2024-05-15T12:40:33.658 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:33 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:33.305+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:34.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:34 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:34.334+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6806 osd.0 since back 2024-05-15T12:39:32.671360+0000 front 2024-05-15T12:39:32.671184+0000 (oldest deadline 2024-05-15T12:39:59.671652+0000) 2024-05-15T12:40:34.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:34 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:34.334+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6814 osd.1 since back 2024-05-15T12:39:45.473217+0000 front 2024-05-15T12:39:45.473259+0000 (oldest deadline 2024-05-15T12:40:08.973061+0000) 2024-05-15T12:40:34.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:34 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:34.334+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6822 osd.2 since back 2024-05-15T12:39:53.074008+0000 front 2024-05-15T12:39:53.074029+0000 (oldest deadline 2024-05-15T12:40:19.474252+0000) 2024-05-15T12:40:34.744 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:34 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:34.334+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.102:6830 osd.3 since back 2024-05-15T12:40:01.775267+0000 front 2024-05-15T12:40:01.775238+0000 (oldest deadline 2024-05-15T12:40:26.475092+0000) 2024-05-15T12:40:34.745 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:34 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:34.334+0000 7f598f06a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.173:6804 osd.4 since back 2024-05-15T12:40:06.475566+0000 front 2024-05-15T12:40:06.475500+0000 (oldest deadline 2024-05-15T12:40:31.775599+0000) 2024-05-15T12:40:34.745 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:34 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7[44132]: 2024-05-15T12:40:34.334+0000 7f598f06a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14547.0:148 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-05-15T12:40:35.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:35 smithi173 podman[62687]: 2024-05-15 12:40:35.242284054 +0000 UTC m=+5.456517936 container died f46ffc322b3f8d24f98b50e1f222b50742584650101ebe99f7563b356ba78fa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240514, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-05-15T12:40:35.862 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:35 smithi173 podman[62687]: 2024-05-15 12:40:35.671428711 +0000 UTC m=+5.885662615 container cleanup f46ffc322b3f8d24f98b50e1f222b50742584650101ebe99f7563b356ba78fa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7, org.label-schema.build-date=20240514, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-15T12:40:35.862 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:35 smithi173 bash[62687]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7 2024-05-15T12:40:36.200 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:35 smithi173 podman[62715]: 2024-05-15 12:40:35.872694427 +0000 UTC m=+0.620145910 container remove f46ffc322b3f8d24f98b50e1f222b50742584650101ebe99f7563b356ba78fa5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-osd-ec-partial-reads-ae24812) 2024-05-15T12:40:36.494 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:36 smithi173 podman[62746]: 2024-05-15 12:40:36.401064716 +0000 UTC m=+0.412274633 container died 97c8ecf187a7658e6e2e57be55eba8743434fef5d7bf788a051cb4539a5f8ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=romantic_carver, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-15T12:40:37.983 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:37 smithi173 podman[62868]: 2024-05-15 12:40:37.630800462 +0000 UTC m=+0.017428969 image pull bce43dd9c04a5d1efc353fa6a8cab31ce3dda72b4628285173869789664d92a9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328 2024-05-15T12:40:37.984 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:37 smithi173 podman[62868]: 2024-05-15 12:40:37.756637379 +0000 UTC m=+0.143265903 container create 5cf442c740d4d7bfa2e01be36ef64ea64d87693885c903ba6c58da67d0f50ba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-osd-ec-partial-reads-ae24812) 2024-05-15T12:40:37.984 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:37 smithi173 podman[62868]: 2024-05-15 12:40:37.993944682 +0000 UTC m=+0.380573188 container init 5cf442c740d4d7bfa2e01be36ef64ea64d87693885c903ba6c58da67d0f50ba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240514) 2024-05-15T12:40:38.244 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:37 smithi173 podman[62868]: 2024-05-15 12:40:37.99791355 +0000 UTC m=+0.384542052 container start 5cf442c740d4d7bfa2e01be36ef64ea64d87693885c903ba6c58da67d0f50ba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240514, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-15T12:40:38.244 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:38 smithi173 podman[62868]: 2024-05-15 12:40:38.069237319 +0000 UTC m=+0.455865837 container attach 5cf442c740d4d7bfa2e01be36ef64ea64d87693885c903ba6c58da67d0f50ba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812, ceph=True, org.label-schema.build-date=20240514, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-15T12:40:38.729 INFO:journalctl@ceph.osd.7.smithi173.stdout:May 15 12:40:38 smithi173 podman[62868]: 2024-05-15 12:40:38.487169254 +0000 UTC m=+0.873797756 container died 5cf442c740d4d7bfa2e01be36ef64ea64d87693885c903ba6c58da67d0f50ba7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240514, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-05-15T12:40:39.527 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@osd.7.service' 2024-05-15T12:40:39.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:40:39.574 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-15T12:40:39.574 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-05-15T12:40:39.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@rgw.foo.a 2024-05-15T12:40:39.872 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:40:39 smithi102 systemd[1]: Stopping Ceph rgw.foo.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:40.152 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:40:39 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a[50136]: 2024-05-15T12:40:39.881+0000 7f82bcfde700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-15T12:40:40.152 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:40:39 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a[50136]: 2024-05-15T12:40:39.881+0000 7f82cb857ac0 -1 shutting down 2024-05-15T12:40:41.903 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:40:41.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi102.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.102:8443: connect: connection refused" 2024-05-15T12:40:41.903 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:41 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:40:41.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-05-15T12:40:50.153 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:40:49 smithi102 bash[73924]: time="2024-05-15T12:40:49Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-05-15T12:40:50.153 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:40:49 smithi102 podman[73924]: 2024-05-15 12:40:49.90353894 +0000 UTC m=+10.117424067 container died cd31bb0d8a84cc8297a52c989fae94b4f22a8601e640fac3bac9c07d6ade8f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-15T12:40:50.522 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:40:50 smithi102 podman[73924]: 2024-05-15 12:40:50.306649525 +0000 UTC m=+10.520534650 container cleanup cd31bb0d8a84cc8297a52c989fae94b4f22a8601e640fac3bac9c07d6ade8f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a, org.label-schema.build-date=20240514, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-osd-ec-partial-reads-ae24812, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-05-15T12:40:50.522 INFO:journalctl@ceph.rgw.foo.a.smithi102.stdout:May 15 12:40:50 smithi102 bash[73924]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-rgw-foo-a 2024-05-15T12:40:50.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@rgw.foo.a.service' 2024-05-15T12:40:50.714 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:40:50.714 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-05-15T12:40:50.714 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-05-15T12:40:50.714 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@prometheus.a 2024-05-15T12:40:50.994 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:50 smithi173 systemd[1]: Stopping Ceph prometheus.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:51.279 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-05-15T12:40:51.279 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.023Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.024Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-05-15T12:40:51.280 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a[59894]: ts=2024-05-15T12:40:51.024Z caller=main.go:1147 level=info msg="See you next time!" 2024-05-15T12:40:51.281 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 podman[63708]: 2024-05-15 12:40:51.033902383 +0000 UTC m=+0.105470665 container died 36f44fe93bc7afda148206c721d4dac9b47fd411bfa6d7c034e7137d31446267 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:51.744 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 podman[63708]: 2024-05-15 12:40:51.481556286 +0000 UTC m=+0.553124553 container cleanup 36f44fe93bc7afda148206c721d4dac9b47fd411bfa6d7c034e7137d31446267 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:51.744 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:May 15 12:40:51 smithi173 bash[63708]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-prometheus-a 2024-05-15T12:40:51.809 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:51 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:40:51.625Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi102.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.102:8443: connect: connection refused" 2024-05-15T12:40:51.809 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:51 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:40:51.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi102.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.102:8443: connect: connection refused" 2024-05-15T12:40:51.810 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:51 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:40:51.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi173.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.173:8443: connect: connection refused" 2024-05-15T12:40:51.910 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@prometheus.a.service' 2024-05-15T12:40:51.945 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-15T12:40:51.945 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-05-15T12:40:51.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-15T12:40:54.120 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:40:56.152 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:56 smithi102 systemd[1]: Stopping Ceph alertmanager.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:56.572 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:56 smithi102 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a[70011]: ts=2024-05-15T12:40:56.315Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-05-15T12:40:56.572 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:56 smithi102 podman[75205]: 2024-05-15 12:40:56.336318311 +0000 UTC m=+0.118803375 container died 6c37c0bb2b5ec344861ecd34baccd5f69c1c5c54e67d5f7b68ac86a41874ad3a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:56.864 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:56 smithi102 podman[75205]: 2024-05-15 12:40:56.673795181 +0000 UTC m=+0.456280248 container cleanup 6c37c0bb2b5ec344861ecd34baccd5f69c1c5c54e67d5f7b68ac86a41874ad3a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:56.864 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:56 smithi102 bash[75205]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a 2024-05-15T12:40:57.152 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:56 smithi102 podman[75219]: 2024-05-15 12:40:56.874953714 +0000 UTC m=+0.538581645 container remove 6c37c0bb2b5ec344861ecd34baccd5f69c1c5c54e67d5f7b68ac86a41874ad3a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:57.152 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:56 smithi102 podman[75219]: 2024-05-15 12:40:56.959035684 +0000 UTC m=+0.622663614 volume remove db10c4a6ae84640f7081dbe48431244f0df1d001f0d1e1c91c68424cc359902c 2024-05-15T12:40:57.653 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:57 smithi102 systemd[1]: Stopping Ceph node-exporter.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:40:57.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:57 smithi102 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-05-15T12:40:57.653 INFO:journalctl@ceph.alertmanager.a.smithi102.stdout:May 15 12:40:57 smithi102 systemd[1]: Stopped Ceph alertmanager.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:40:57.978 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:57 smithi102 podman[75331]: 2024-05-15 12:40:57.716240693 +0000 UTC m=+0.128171921 container died 3ec530770871f0cfeaf21b78ee51a2f12e8904e2d2bcfc5514645deb4ceadfd6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:58.344 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:58 smithi102 podman[75331]: 2024-05-15 12:40:58.086864354 +0000 UTC m=+0.498795573 container cleanup 3ec530770871f0cfeaf21b78ee51a2f12e8904e2d2bcfc5514645deb4ceadfd6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:58.344 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:58 smithi102 bash[75331]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a 2024-05-15T12:40:58.652 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:58 smithi102 podman[75343]: 2024-05-15 12:40:58.354638091 +0000 UTC m=+0.640350962 container remove 3ec530770871f0cfeaf21b78ee51a2f12e8904e2d2bcfc5514645deb4ceadfd6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-05-15T12:40:58.652 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:58 smithi102 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-05-15T12:40:58.652 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:58 smithi102 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-05-15T12:40:58.653 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:58 smithi102 systemd[1]: Stopped Ceph node-exporter.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:40:58.653 INFO:journalctl@ceph.node-exporter.a.smithi102.stdout:May 15 12:40:58 smithi102 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.a.service: Consumed 1.853s CPU time. 2024-05-15T12:40:59.147 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-15T12:40:59.939 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:41:01.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:01 smithi173 systemd[1]: Stopping Ceph iscsi.iscsi.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:41:02.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:01 smithi173 ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a[46619]: debug Shutdown received 2024-05-15T12:41:12.089 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:11 smithi173 bash[64309]: time="2024-05-15T12:41:11Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-05-15T12:41:12.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:11 smithi173 podman[64309]: 2024-05-15 12:41:11.823932411 +0000 UTC m=+10.129004381 container died 0365b95a94001275ca54202c28b58979bb4661d9f4c9cc03d4eda55ebe0a70f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240514, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-osd-ec-partial-reads-ae24812, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-15T12:41:12.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:12 smithi173 podman[64309]: 2024-05-15 12:41:12.265408721 +0000 UTC m=+10.570480688 container cleanup 0365b95a94001275ca54202c28b58979bb4661d9f4c9cc03d4eda55ebe0a70f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240514, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-15T12:41:12.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:12 smithi173 bash[64309]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a 2024-05-15T12:41:12.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:12 smithi173 podman[64322]: 2024-05-15 12:41:12.441643955 +0000 UTC m=+0.617023658 container remove 0365b95a94001275ca54202c28b58979bb4661d9f4c9cc03d4eda55ebe0a70f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:900ea3b54c0d576c130cc441649061bec9216d6b88e6f3fb2d03fcff884a4328, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-osd-ec-partial-reads-ae24812, org.label-schema.license=GPLv2, org.label-schema.build-date=20240514) 2024-05-15T12:41:12.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:12 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-05-15T12:41:12.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:12 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-05-15T12:41:12.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:12 smithi173 systemd[1]: Stopped Ceph iscsi.iscsi.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:41:12.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:May 15 12:41:12 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@iscsi.iscsi.a.service: Consumed 2.941s CPU time. 2024-05-15T12:41:14.466 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:14 smithi173 systemd[1]: Stopping Ceph grafana.a for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:41:14.738 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:14 smithi173 podman[64587]: 2024-05-15 12:41:14.476390891 +0000 UTC m=+0.100273896 container died 2e7edb6cbbab7299d2d23e9e141eff85ad86a63fc677c3d8dce98169982aeb1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a, distribution-scope=public, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., io.buildah.version=1.32.0, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-05-15T12:41:14.994 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:14 smithi173 podman[64587]: 2024-05-15 12:41:14.823823291 +0000 UTC m=+0.447706296 container cleanup 2e7edb6cbbab7299d2d23e9e141eff85ad86a63fc677c3d8dce98169982aeb1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.32.0, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, vendor=Red Hat, Inc., description=Ceph Grafana Container, distribution-scope=public) 2024-05-15T12:41:14.995 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:14 smithi173 bash[64587]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a 2024-05-15T12:41:15.437 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:15 smithi173 podman[64601]: 2024-05-15 12:41:15.091410344 +0000 UTC m=+0.617199693 container remove 2e7edb6cbbab7299d2d23e9e141eff85ad86a63fc677c3d8dce98169982aeb1c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, description=Ceph Grafana Container, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, release=239.1651231664, vendor=Red Hat, Inc., version=8.5, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897) 2024-05-15T12:41:15.437 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:15 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-05-15T12:41:15.437 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:15 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-05-15T12:41:15.437 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:15 smithi173 systemd[1]: Stopped Ceph grafana.a for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:41:15.438 INFO:journalctl@ceph.grafana.a.smithi173.stdout:May 15 12:41:15 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@grafana.a.service: Consumed 4.561s CPU time. 2024-05-15T12:41:15.702 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:15 smithi173 systemd[1]: Stopping Ceph node-exporter.b for eac36fa0-12b6-11ef-bc98-c7b262605968... 2024-05-15T12:41:15.959 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:15 smithi173 podman[64714]: 2024-05-15 12:41:15.713446611 +0000 UTC m=+0.126224715 container died 2ce0d05c5eebe5c797a8e1894cb3b0fa2f269d34082478a2dc895b36bf3ed278 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-15T12:41:16.244 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:16 smithi173 podman[64714]: 2024-05-15 12:41:16.152858789 +0000 UTC m=+0.565636884 container cleanup 2ce0d05c5eebe5c797a8e1894cb3b0fa2f269d34082478a2dc895b36bf3ed278 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-15T12:41:16.244 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:16 smithi173 bash[64714]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b 2024-05-15T12:41:16.708 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:16 smithi173 podman[64728]: 2024-05-15 12:41:16.445851585 +0000 UTC m=+0.736381955 container remove 2ce0d05c5eebe5c797a8e1894cb3b0fa2f269d34082478a2dc895b36bf3ed278 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eac36fa0-12b6-11ef-bc98-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-05-15T12:41:16.708 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:16 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-05-15T12:41:16.709 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:16 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-05-15T12:41:16.709 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:16 smithi173 systemd[1]: Stopped Ceph node-exporter.b for eac36fa0-12b6-11ef-bc98-c7b262605968. 2024-05-15T12:41:16.709 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:May 15 12:41:16 smithi173 systemd[1]: ceph-eac36fa0-12b6-11ef-bc98-c7b262605968@node-exporter.b.service: Consumed 1.796s CPU time. 2024-05-15T12:41:17.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:41:17.235 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-15T12:41:17.262 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-15T12:41:17.264 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113/remote/smithi102/crash 2024-05-15T12:41:17.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/crash -- . 2024-05-15T12:41:17.318 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T12:41:17.319 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-05-15T12:41:17.321 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113/remote/smithi173/crash 2024-05-15T12:41:17.322 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/crash -- . 2024-05-15T12:41:17.356 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-15T12:41:17.356 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-05-15T12:41:17.357 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-15T12:41:17.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-05-15T12:41:17.392 INFO:tasks.cephadm:Compressing logs... 2024-05-15T12:41:17.392 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T12:41:17.435 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-15T12:41:17.460 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-15T12:41:17.460 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-05-15T12:41:17.460 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.log 2024-05-15T12:41:17.460 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-15T12:41:17.460 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-15T12:41:17.461 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-15T12:41:17.461 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-15T12:41:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-15T12:41:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:: No such file or directory 2024-05-15T12:41:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.log 2024-05-15T12:41:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-15T12:41:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-15T12:41:17.462 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.log: /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-15T12:41:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose --/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.audit.log: /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-15T12:41:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-15T12:41:17.466 INFO:teuthology.orchestra.run.smithi102.stderr: 91.9% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-15T12:41:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-15T12:41:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.1.log: /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-15T12:41:17.466 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-15T12:41:17.467 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.7.log: 85.9% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-15T12:41:17.467 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-15T12:41:17.467 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-15T12:41:17.467 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.4.log: /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/tcmu-runner.log 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr: 90.3% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/tcmu-runner.log: 67.4% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/tcmu-runner.log.gz 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr: 81.6% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.b.log: 82.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T12:41:17.468 INFO:teuthology.orchestra.run.smithi173.stderr: 90.7% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-15T12:41:17.470 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.c.log: 93.9% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-15T12:41:17.470 INFO:teuthology.orchestra.run.smithi102.stderr: 86.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-15T12:41:17.470 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-15T12:41:17.470 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-15T12:41:17.471 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-15T12:41:17.471 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-client.rgw.foo.a.log 2024-05-15T12:41:17.482 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-client.rgw.foo.a.log: 57.8% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-05-15T12:41:17.502 INFO:teuthology.orchestra.run.smithi173.stderr: 90.2% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-15T12:41:17.531 INFO:teuthology.orchestra.run.smithi102.stderr: 90.2% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-15T12:41:17.632 INFO:teuthology.orchestra.run.smithi173.stderr: 92.1% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-15T12:41:17.633 INFO:teuthology.orchestra.run.smithi102.stderr: 88.2% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-15T12:41:17.643 INFO:teuthology.orchestra.run.smithi102.stderr: 92.3% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-15T12:41:17.655 INFO:teuthology.orchestra.run.smithi173.stderr: 93.1% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-15T12:41:17.670 INFO:teuthology.orchestra.run.smithi102.stderr: 92.9% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-15T12:41:17.671 INFO:teuthology.orchestra.run.smithi173.stderr: 92.8% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-15T12:41:17.676 INFO:teuthology.orchestra.run.smithi173.stderr: 93.0% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-15T12:41:17.699 INFO:teuthology.orchestra.run.smithi173.stderr: 93.4% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-15T12:41:17.701 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-05-15T12:41:17.701 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m0.251s 2024-05-15T12:41:17.701 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m1.042s 2024-05-15T12:41:17.701 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.063s 2024-05-15T12:41:17.716 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-15T12:41:17.754 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-15T12:41:17.787 INFO:teuthology.orchestra.run.smithi102.stderr: 93.5% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-15T12:41:17.795 INFO:teuthology.orchestra.run.smithi102.stderr: 91.2% -- replaced with /var/log/ceph/eac36fa0-12b6-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-15T12:41:17.798 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-05-15T12:41:17.798 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m0.348s 2024-05-15T12:41:17.798 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m1.763s 2024-05-15T12:41:17.798 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.090s 2024-05-15T12:41:17.798 INFO:tasks.cephadm:Archiving logs... 2024-05-15T12:41:17.799 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113/remote/smithi102/log 2024-05-15T12:41:17.800 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T12:41:18.084 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113/remote/smithi173/log 2024-05-15T12:41:18.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-15T12:41:18.251 INFO:tasks.cephadm:Removing cluster... 2024-05-15T12:41:18.252 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 --force 2024-05-15T12:41:18.435 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:41:18.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eac36fa0-12b6-11ef-bc98-c7b262605968 --force 2024-05-15T12:41:18.751 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: eac36fa0-12b6-11ef-bc98-c7b262605968 2024-05-15T12:41:18.873 INFO:tasks.cephadm:Removing cephadm ... 2024-05-15T12:41:18.873 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T12:41:18.888 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-15T12:41:18.904 INFO:tasks.cephadm:Teardown complete 2024-05-15T12:41:18.904 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-05-15T12:41:18.914 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_1... 2024-05-15T12:41:18.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_1 2024-05-15T12:41:19.067 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-05-15T12:41:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:19.068 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_2... 2024-05-15T12:41:19.069 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_2 2024-05-15T12:41:19.234 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-05-15T12:41:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:19.236 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_3... 2024-05-15T12:41:19.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_3 2024-05-15T12:41:19.370 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-05-15T12:41:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:19.372 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi102:/dev/vg_nvme/lv_4... 2024-05-15T12:41:19.372 DEBUG:teuthology.orchestra.run.smithi102:> sudo nvme disconnect -n lv_4 2024-05-15T12:41:19.511 INFO:teuthology.orchestra.run.smithi102.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-05-15T12:41:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:19.512 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-05-15T12:41:19.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/scratch_devs 2024-05-15T12:41:19.537 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi173:/dev/vg_nvme/lv_1... 2024-05-15T12:41:19.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo nvme disconnect -n lv_1 2024-05-15T12:41:19.680 INFO:teuthology.orchestra.run.smithi173.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-05-15T12:41:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:19.682 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi173:/dev/vg_nvme/lv_2... 2024-05-15T12:41:19.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo nvme disconnect -n lv_2 2024-05-15T12:41:19.830 INFO:teuthology.orchestra.run.smithi173.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-05-15T12:41:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:19.832 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi173:/dev/vg_nvme/lv_3... 2024-05-15T12:41:19.832 DEBUG:teuthology.orchestra.run.smithi173:> sudo nvme disconnect -n lv_3 2024-05-15T12:41:19.980 INFO:teuthology.orchestra.run.smithi173.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-05-15T12:41:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:19.981 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi173:/dev/vg_nvme/lv_4... 2024-05-15T12:41:19.982 DEBUG:teuthology.orchestra.run.smithi173:> sudo nvme disconnect -n lv_4 2024-05-15T12:41:20.128 INFO:teuthology.orchestra.run.smithi173.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-05-15T12:41:20.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:20.130 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-05-15T12:41:20.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/scratch_devs 2024-05-15T12:41:20.159 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-15T12:41:20.168 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-15T12:41:20.168 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:41:20.170 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-15T12:41:20.185 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:41:20.190 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:41:20.190 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-05-15T12:41:20.190 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +1792us[+1799us] +/- 43ms 2024-05-15T12:41:20.190 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 65 -1409us[-1403us] +/- 60ms 2024-05-15T12:41:20.190 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 -1034us[-1034us] +/- 67ms 2024-05-15T12:41:20.190 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:41:20.217 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-05-15T12:41:20.222 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-15T12:41:20.222 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-05-15T12:41:20.222 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +1792us[+1802us] +/- 43ms 2024-05-15T12:41:20.223 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -1423us[-1423us] +/- 61ms 2024-05-15T12:41:20.223 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 -1061us[-1051us] +/- 66ms 2024-05-15T12:41:20.223 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-15T12:41:20.223 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-15T12:41:20.232 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-15T12:41:20.232 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-15T12:41:20.275 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T12:41:20.348 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-15T12:41:20.422 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:41:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:20.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-05-15T12:41:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:20.482 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-15T12:41:20.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-15T12:41:20.516 INFO:teuthology.task.internal:Duration was 910.566700 seconds 2024-05-15T12:41:20.517 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-15T12:41:20.566 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-15T12:41:20.566 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T12:41:20.569 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-15T12:41:20.606 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:41:20.609 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-15T12:41:20.936 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-15T12:41:20.936 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-05-15T12:41:20.936 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T12:41:20.997 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-05-15T12:41:20.998 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-15T12:41:21.021 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-15T12:41:21.022 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T12:41:21.040 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-15T12:41:21.089 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-15T12:41:21.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T12:41:21.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-15T12:41:21.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-15T12:41:21.379 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-15T12:41:21.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T12:41:21.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-15T12:41:21.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-15T12:41:21.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:41:21.446 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:41:21.467 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-05-15T12:41:21.492 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-05-15T12:41:21.508 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:41:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:21.550 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-15T12:41:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-15T12:41:21.565 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-15T12:41:21.573 INFO:teuthology.task.internal:Transferring archived files... 2024-05-15T12:41:21.574 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113/remote/smithi102 2024-05-15T12:41:21.574 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T12:41:21.630 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rzarzynski-2024-05-15_11:53:37-rados-wip-osd-ec-partial-reads-distro-default-smithi/7707113/remote/smithi173 2024-05-15T12:41:21.630 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-15T12:41:21.667 INFO:teuthology.task.internal:Removing archive directory... 2024-05-15T12:41:21.667 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T12:41:21.670 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-15T12:41:21.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-15T12:41:21.726 INFO:teuthology.task.internal:Not uploading archives. 2024-05-15T12:41:21.726 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-15T12:41:21.758 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-15T12:41:21.759 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T12:41:21.761 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-15T12:41:21.773 INFO:teuthology.orchestra.run.smithi102.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 12:41 /home/ubuntu/cephtest 2024-05-15T12:41:21.775 INFO:teuthology.orchestra.run.smithi173.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 15 12:41 /home/ubuntu/cephtest 2024-05-15T12:41:21.776 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-15T12:41:21.809 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-15T12:41:21.911 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 910.5667004585266 owner: scheduled_rzarzynski@teuthology success: true 2024-05-15T12:41:21.911 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-15T12:41:21.995 INFO:teuthology.run:pass