2024-03-16T14:16:36.139 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-16T14:16:36.139 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} --archive /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389 --name yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389/orig.config.yaml 2024-03-16T14:16:36.167 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T14:16:36.274 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389 branch: wip-yuri6-testing-2024-03-15-0709 description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} email: yweinste@redhat.com first_in_suite: false job_id: '7603389' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri6-testing-2024-03-15-0709 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN sha1: 5ea4d9cfac4d9f2893126a48d28cca7da69243c1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5ea4d9cfac4d9f2893126a48d28cca7da69243c1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri6-testing-2024-03-15-0709 sha1: 5ea4d9cfac4d9f2893126a48d28cca7da69243c1 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9615 sha1: 5ea4d9cfac4d9f2893126a48d28cca7da69243c1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2024-03-15-0709 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5ea4d9cfac4d9f2893126a48d28cca7da69243c1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5ea4d9cfac4d9f2893126a48d28cca7da69243c1 targets: smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9MNqEvWvI8i3vOiK98ta6iTbCewtR19hA3NMWlr1/+lc9ucxHNazUHUXpAoeJ6QHwPZzRJZ6Tc6lHH/K1Zwhk= smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbcxlXkGa3pIdZPgFfZ8SvggEytJw1aDR29LjeBlt3FeVccg/EmmW/JWPf+qrViJKt9eFTfKFJlcI+V6VU60JM= tasks: - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-15_19:59:43 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-16T14:16:36.275 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5ea4d9cfac4d9f2893126a48d28cca7da69243c1/qa; will attempt to use it 2024-03-16T14:16:36.275 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5ea4d9cfac4d9f2893126a48d28cca7da69243c1/qa/tasks 2024-03-16T14:16:36.275 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-16T14:16:36.277 INFO:teuthology.task.internal:Checking packages... 2024-03-16T14:16:36.301 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5ea4d9cfac4d9f2893126a48d28cca7da69243c1' 2024-03-16T14:16:36.302 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-16T14:16:36.302 INFO:teuthology.packaging:ref: None 2024-03-16T14:16:36.302 INFO:teuthology.packaging:tag: None 2024-03-16T14:16:36.302 INFO:teuthology.packaging:branch: wip-yuri6-testing-2024-03-15-0709 2024-03-16T14:16:36.302 INFO:teuthology.packaging:sha1: 5ea4d9cfac4d9f2893126a48d28cca7da69243c1 2024-03-16T14:16:36.302 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-03-15-0709 2024-03-16T14:16:36.517 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2121.g5ea4d9cf 2024-03-16T14:16:36.518 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-16T14:16:36.525 INFO:teuthology.task.internal:no buildpackages task found 2024-03-16T14:16:36.525 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-16T14:16:36.548 INFO:teuthology.task.internal:Saving configuration 2024-03-16T14:16:36.558 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-16T14:16:36.564 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T14:16:36.587 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 14:11:21.966966', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9MNqEvWvI8i3vOiK98ta6iTbCewtR19hA3NMWlr1/+lc9ucxHNazUHUXpAoeJ6QHwPZzRJZ6Tc6lHH/K1Zwhk='} 2024-03-16T14:16:36.608 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 14:11:21.968921', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbcxlXkGa3pIdZPgFfZ8SvggEytJw1aDR29LjeBlt3FeVccg/EmmW/JWPf+qrViJKt9eFTfKFJlcI+V6VU60JM='} 2024-03-16T14:16:36.608 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-16T14:16:36.616 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-16T14:16:36.616 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-16T14:16:36.616 INFO:teuthology.run_tasks:Running task console_log... 2024-03-16T14:16:36.688 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f68e6998160>, signals=[15]) 2024-03-16T14:16:36.688 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-16T14:16:36.694 INFO:teuthology.task.internal:Opening connections... 2024-03-16T14:16:36.694 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-03-16T14:16:36.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:16:36.840 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-03-16T14:16:36.841 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:16:36.930 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-16T14:16:36.936 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-03-16T14:16:36.954 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-03-16T14:16:36.955 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-03-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-03-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-03-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-03-16T14:16:37.011 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T14:16:37.012 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T14:16:37.013 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-03-16T14:16:37.037 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-03-16T14:16:37.055 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-03-16T14:16:37.055 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-03-16T14:16:37.111 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-03-16T14:16:37.111 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-03-16T14:16:37.111 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-03-16T14:16:37.111 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-03-16T14:16:37.111 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-03-16T14:16:37.111 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-16T14:16:37.112 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-16T14:16:37.112 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-03-16T14:16:37.137 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-16T14:16:37.145 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-16T14:16:37.150 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-16T14:16:37.150 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-03-16T14:16:37.153 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-03-16T14:16:37.169 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-16T14:16:37.175 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-16T14:16:37.175 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-03-16T14:16:37.211 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-03-16T14:16:37.319 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-16T14:16:37.371 INFO:teuthology.run_tasks:Running task kernel... 2024-03-16T14:16:37.386 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-16T14:16:37.386 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-16T14:16:37.386 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T14:16:37.387 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-16T14:16:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:16:37.403 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-16T14:16:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:16:37.404 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-16T14:16:37.460 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-361.el9.x86_64 2024-03-16T14:16:37.460 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-361.el9.x86_64 2024-03-16T14:16:37.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-03-16T14:16:37.462 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-361.el9.x86_64 2024-03-16T14:16:37.462 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-361.el9.x86_64 2024-03-16T14:16:37.462 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-03-16T14:16:39.697 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-16T14:16:39.709 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-16T14:16:41.134 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-16T14:16:41.406 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-16T14:16:45.646 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-16T14:16:48.097 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-16T14:16:51.191 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 847 kB/s | 8.0 MB 00:09 2024-03-16T14:16:51.914 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.1 MB 00:00 2024-03-16T14:16:53.407 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2024-03-16T14:16:54.184 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-16T14:16:56.609 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 21 MB 00:03 2024-03-16T14:17:01.317 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.1 MB 00:03 2024-03-16T14:17:01.566 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 22 kB/s | 1.7 kB 00:00 2024-03-16T14:17:03.127 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 15 kB 00:00 2024-03-16T14:17:03.706 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T14:17:03.707 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T14:17:03.813 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-03-16T14:17:03.828 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-16T14:17:03.828 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-03-16T14:17:03.828 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-16T14:17:03.828 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-03-16T14:17:03.829 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:17:03.830 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 100 M 2024-03-16T14:17:03.830 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 123 M 2024-03-16T14:17:03.830 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-03-16T14:17:04.444 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-16T14:17:04.459 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-03-16T14:17:04.701 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 26 MB/s | 21 MB 00:00 2024-03-16T14:17:05.383 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 26 MB/s | 39 MB 00:01 2024-03-16T14:17:06.093 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 20 MB/s | 33 MB 00:01 2024-03-16T14:17:06.094 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-03-16T14:17:06.094 INFO:teuthology.orchestra.run.smithi151.stdout:Total 44 MB/s | 100 MB 00:02 2024-03-16T14:17:06.977 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-03-16T14:17:07.373 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-03-16T14:17:07.373 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-03-16T14:17:08.733 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-03-16T14:17:08.733 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-03-16T14:17:09.553 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-16T14:17:11.454 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-03-16T14:17:11.851 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-16T14:17:11.851 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-16T14:17:11.942 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-16T14:17:11.952 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-16T14:17:11.952 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-16T14:17:11.952 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-16T14:17:11.952 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-03-16T14:17:11.953 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:17:11.954 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 100 M 2024-03-16T14:17:11.954 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 123 M 2024-03-16T14:17:11.954 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-16T14:17:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T14:17:12.261 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T14:17:13.762 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.4 MB/s | 6.0 MB 00:01 2024-03-16T14:17:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T14:17:15.304 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T14:17:15.704 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 5.7 MB/s | 21 MB 00:03 2024-03-16T14:17:17.562 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 8.8 MB/s | 33 MB 00:03 2024-03-16T14:17:19.371 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 5.3 MB/s | 39 MB 00:07 2024-03-16T14:17:19.371 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-16T14:17:19.371 INFO:teuthology.orchestra.run.smithi026.stdout:Total 13 MB/s | 100 MB 00:07 2024-03-16T14:17:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T14:17:20.181 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:20.291 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-16T14:17:20.725 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-16T14:17:20.725 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-16T14:17:22.123 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-16T14:17:22.123 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-16T14:17:24.954 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:24.970 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-16T14:17:25.686 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-16T14:17:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T14:17:28.741 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T14:17:29.064 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T14:17:34.165 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T14:17:34.293 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:39.419 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:54.351 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:54.352 INFO:teuthology.orchestra.run.smithi151.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T14:17:54.352 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:17:54.437 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:54.870 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:54.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T14:17:54.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T14:17:54.870 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T14:17:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:17:55.279 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:17:55.279 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-03-16T14:17:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T14:17:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T14:17:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T14:17:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T14:17:55.280 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:17:55.280 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-03-16T14:17:55.787 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T14:17:55.926 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T14:17:55.927 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T14:17:55.927 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T14:17:55.927 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T14:17:55.927 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T14:17:55.927 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-16T14:17:55.927 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi151.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T14:17:55.927 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T14:17:56.049 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T14:17:56.050 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T14:17:56.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo rpm -qi grub2-tools 2024-03-16T14:17:56.087 INFO:teuthology.orchestra.run.smithi151.stdout:Name : grub2-tools 2024-03-16T14:17:56.087 INFO:teuthology.orchestra.run.smithi151.stdout:Epoch : 1 2024-03-16T14:17:56.087 INFO:teuthology.orchestra.run.smithi151.stdout:Version : 2.06 2024-03-16T14:17:56.087 INFO:teuthology.orchestra.run.smithi151.stdout:Release : 68.el9 2024-03-16T14:17:56.087 INFO:teuthology.orchestra.run.smithi151.stdout:Architecture: x86_64 2024-03-16T14:17:56.087 INFO:teuthology.orchestra.run.smithi151.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Group : Unspecified 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Size : 8264254 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:License : GPLv3+ 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Packager : builder@centos.org 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Vendor : CentOS 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Summary : Support tools for GRUB. 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:Description : 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:17:56.088 INFO:teuthology.orchestra.run.smithi151.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T14:17:56.089 INFO:teuthology.orchestra.run.smithi151.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T14:17:56.089 INFO:teuthology.orchestra.run.smithi151.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T14:17:56.089 INFO:teuthology.orchestra.run.smithi151.stdout:hardware devices. 2024-03-16T14:17:56.089 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:17:56.089 INFO:teuthology.orchestra.run.smithi151.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T14:17:56.090 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T14:17:56.090 INFO:teuthology.task.kernel:Updating grub on smithi151 to boot 5.14.0-427.el9.x86_64 2024-03-16T14:17:56.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T14:17:56.664 INFO:teuthology.orchestra.run.smithi151.stderr:Generating grub configuration file ... 2024-03-16T14:17:58.041 INFO:teuthology.orchestra.run.smithi151.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T14:17:58.074 INFO:teuthology.orchestra.run.smithi151.stderr:done 2024-03-16T14:17:58.077 DEBUG:teuthology.orchestra.run.smithi151:> mktemp 2024-03-16T14:17:58.093 INFO:teuthology.orchestra.run.smithi151.stdout:/tmp/tmp.SPceFcP1n4 2024-03-16T14:17:58.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SPceFcP1n4 2024-03-16T14:17:58.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0666 /tmp/tmp.SPceFcP1n4 2024-03-16T14:17:58.306 DEBUG:teuthology.orchestra.remote:smithi151:/tmp/tmp.SPceFcP1n4 is 6KB 2024-03-16T14:17:58.326 DEBUG:teuthology.orchestra.run.smithi151:> rm -fr /tmp/tmp.SPceFcP1n4 2024-03-16T14:17:58.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T14:17:58.405 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T14:17:58.406 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T14:17:58.406 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T14:17:58.406 INFO:teuthology.orchestra.run.smithi151.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T14:17:58.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T14:17:58.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo shutdown -r now 2024-03-16T14:18:10.108 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:18:10.109 INFO:teuthology.orchestra.run.smithi026.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-16T14:18:10.109 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:18:10.196 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:18:10.757 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:18:10.757 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-16T14:18:10.757 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-16T14:18:10.757 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-16T14:18:11.202 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-16T14:18:11.202 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:18:11.202 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-16T14:18:11.202 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-16T14:18:11.202 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-16T14:18:11.202 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-16T14:18:11.203 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-16T14:18:11.203 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:18:11.203 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-16T14:18:11.734 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T14:18:11.887 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T14:18:11.887 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T14:18:11.887 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-16T14:18:11.887 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-16T14:18:11.887 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-16T14:18:11.887 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-16T14:18:11.887 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-03-16T14:18:11.887 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-16T14:18:12.012 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-16T14:18:12.013 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-16T14:18:12.013 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-03-16T14:18:12.058 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-03-16T14:18:12.058 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-03-16T14:18:12.058 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-03-16T14:18:12.058 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 68.el9 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8264254 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:18:12.059 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-16T14:18:12.060 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-16T14:18:12.060 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-16T14:18:12.060 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-03-16T14:18:12.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:18:12.060 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-03-16T14:18:12.061 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-16T14:18:12.061 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 5.14.0-427.el9.x86_64 2024-03-16T14:18:12.061 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-16T14:18:12.642 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-03-16T14:18:14.108 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-16T14:18:14.146 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-03-16T14:18:14.148 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-16T14:18:14.163 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.lemkhxITHs 2024-03-16T14:18:14.163 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lemkhxITHs 2024-03-16T14:18:14.240 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.lemkhxITHs 2024-03-16T14:18:14.391 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.lemkhxITHs is 6KB 2024-03-16T14:18:14.440 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.lemkhxITHs 2024-03-16T14:18:14.455 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-03-16T14:18:14.524 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-16T14:18:14.524 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-16T14:18:14.524 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-16T14:18:14.524 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-16T14:18:14.525 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-16T14:18:14.695 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-03-16T14:18:28.593 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-16T14:18:28.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-16T14:18:28.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:18:44.703 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-16T14:18:44.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-16T14:18:44.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:18:46.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-16T14:18:55.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-16T14:18:55.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:18:59.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.151 2024-03-16T14:19:03.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-16T14:19:11.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-16T14:19:11.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:19:12.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-16T14:19:12.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:19:30.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-16T14:19:42.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-16T14:19:42.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:19:42.769 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-16T14:19:43.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-16T14:19:43.316 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-16T14:19:43.317 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-16T14:19:43.330 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-427.el9.x86_64 2024-03-16T14:19:43.330 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T14:19:43.330 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T14:19:43.331 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T14:19:44.331 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-16T14:19:44.332 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T14:19:44.425 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-03-16T14:19:44.466 DEBUG:teuthology.parallel:result is None 2024-03-16T14:20:11.017 DEBUG:teuthology.orchestra.remote:timed out 2024-03-16T14:20:26.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-16T14:20:26.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:20:26.429 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-16T14:20:26.820 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-16T14:20:26.820 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-16T14:20:26.821 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-03-16T14:20:26.836 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-427.el9.x86_64 2024-03-16T14:20:26.836 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-16T14:20:26.836 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-16T14:20:26.836 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-16T14:20:27.837 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-16T14:20:27.837 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-16T14:20:27.921 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-03-16T14:20:27.952 DEBUG:teuthology.parallel:result is None 2024-03-16T14:20:27.952 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-16T14:20:27.961 INFO:teuthology.task.internal:Creating test directory... 2024-03-16T14:20:27.961 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T14:20:27.965 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-16T14:20:27.982 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-16T14:20:27.996 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-16T14:20:28.002 INFO:teuthology.task.internal:Creating archive directory... 2024-03-16T14:20:28.002 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T14:20:28.024 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-16T14:20:28.051 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-16T14:20:28.058 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-16T14:20:28.059 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T14:20:28.084 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-16T14:20:28.119 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T14:20:28.131 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T14:20:28.133 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T14:20:28.145 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-16T14:20:28.147 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-16T14:20:28.153 INFO:teuthology.task.internal:Configuring sudo... 2024-03-16T14:20:28.154 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T14:20:28.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-16T14:20:28.207 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-16T14:20:28.216 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-16T14:20:28.216 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T14:20:28.247 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-16T14:20:28.263 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T14:20:28.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T14:20:28.399 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T14:20:28.481 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T14:20:28.526 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:20:28.526 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T14:20:28.595 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T14:20:28.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-16T14:20:28.701 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T14:20:28.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-16T14:20:28.788 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:20:28.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-16T14:20:28.858 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-03-16T14:20:28.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-03-16T14:20:28.909 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T14:20:28.936 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T14:20:29.297 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-16T14:20:29.305 INFO:teuthology.task.internal:Starting timer... 2024-03-16T14:20:29.305 INFO:teuthology.run_tasks:Running task pcp... 2024-03-16T14:20:29.314 INFO:teuthology.run_tasks:Running task selinux... 2024-03-16T14:20:29.322 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-16T14:20:29.323 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-03-16T14:20:29.403 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-03-16T14:20:29.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-03-16T14:20:29.458 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-03-16T14:20:29.460 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-16T14:20:29.460 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-03-16T14:20:29.481 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-03-16T14:20:29.481 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-03-16T14:20:29.512 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-03-16T14:20:29.512 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi026.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi151.front.sepia.ceph.com': 'permissive'} 2024-03-16T14:20:29.512 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-16T14:20:29.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:20:29.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-16T14:20:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:20:29.579 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-16T14:20:29.579 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-03-16T14:20:29.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-03-16T14:20:29.654 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-16T14:20:29.663 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-16T14:20:29.680 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-16T14:20:29.681 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi026.front.sepia.ceph.com,smithi151.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-16T14:24:41.332 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi026.front.sepia.ceph.com'), Remote(name='ubuntu@smithi151.front.sepia.ceph.com')] 2024-03-16T14:24:41.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-16T14:24:41.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:24:41.426 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-16T14:24:41.501 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-16T14:24:41.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-16T14:24:41.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-16T14:24:41.589 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-03-16T14:24:41.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-03-16T14:24:41.669 INFO:teuthology.run_tasks:Running task clock... 2024-03-16T14:24:41.680 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-16T14:24:41.680 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T14:24:41.680 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T14:24:41.682 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-16T14:24:41.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T14:24:41.712 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T14:24:41.729 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T14:24:41.740 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-16T14:24:41.757 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-16T14:24:41.759 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-03-16T14:24:41.773 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-03-16T14:24:41.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-03-16T14:24:41.789 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T14:24:41.799 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-03-16T14:24:41.805 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T14:24:41.816 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-16T14:24:41.832 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-16T14:24:41.856 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-16T14:24:41.860 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T14:24:41.860 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-16T14:24:41.860 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.860 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.860 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.860 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.881 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-16T14:24:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T14:24:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-16T14:24:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.884 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:24:41.885 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-16T14:24:41.895 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-16T14:24:41.896 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:24:41.896 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T14:24:41.915 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T14:24:41.915 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-03-16T14:24:41.969 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T14:24:41.969 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:41.969 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-16T14:24:41.970 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:41.970 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:41.970 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:24:40.310142581 +0000 2024-03-16T14:24:41.970 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:19.567451201 +0000 2024-03-16T14:24:41.970 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:19.567451201 +0000 2024-03-16T14:24:41.970 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:19.567451201 +0000 2024-03-16T14:24:41.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T14:24:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:24:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:24:42.034 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.0354e-05 s, 6.4 MB/s 2024-03-16T14:24:42.035 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T14:24:42.091 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:24:40.310142581 +0000 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:19.902446217 +0000 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:19.902446217 +0000 2024-03-16T14:24:42.146 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:19.902446217 +0000 2024-03-16T14:24:42.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T14:24:42.211 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:24:42.211 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:24:42.211 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.4617e-05 s, 5.4 MB/s 2024-03-16T14:24:42.212 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T14:24:42.267 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-03-16T14:24:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T14:24:42.321 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:42.321 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-16T14:24:42.322 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:42.322 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:42.322 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:24:40.310142581 +0000 2024-03-16T14:24:42.322 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:20.232441307 +0000 2024-03-16T14:24:42.322 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:20.232441307 +0000 2024-03-16T14:24:42.322 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:20.232441307 +0000 2024-03-16T14:24:42.322 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T14:24:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:24:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:24:42.386 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.9467e-05 s, 5.7 MB/s 2024-03-16T14:24:42.387 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T14:24:42.442 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-03-16T14:24:42.496 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T14:24:42.496 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:42.496 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-16T14:24:42.497 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:42.497 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:42.497 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:24:40.310142581 +0000 2024-03-16T14:24:42.497 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:20.577436174 +0000 2024-03-16T14:24:42.497 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:20.577436174 +0000 2024-03-16T14:24:42.497 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:20.577436174 +0000 2024-03-16T14:24:42.497 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T14:24:42.561 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:24:42.561 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:24:42.562 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 8.9618e-05 s, 5.7 MB/s 2024-03-16T14:24:42.563 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T14:24:42.618 DEBUG:teuthology.orchestra.run.smithi026:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-16T14:24:42.757 INFO:teuthology.orchestra.run.smithi026.stdout:loop 2024-03-16T14:24:42.759 INFO:tasks.nvme_loop:Connecting nvme_loop smithi026:/dev/vg_nvme/lv_1... 2024-03-16T14:24:42.759 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-16T14:24:42.796 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-03-16T14:24:42.833 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/vg_nvme/lv_11 2024-03-16T14:24:42.862 INFO:tasks.nvme_loop:Connecting nvme_loop smithi026:/dev/vg_nvme/lv_2... 2024-03-16T14:24:42.863 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-16T14:24:42.947 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-03-16T14:24:42.984 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/vg_nvme/lv_21 2024-03-16T14:24:43.019 INFO:tasks.nvme_loop:Connecting nvme_loop smithi026:/dev/vg_nvme/lv_3... 2024-03-16T14:24:43.019 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-16T14:24:43.097 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-03-16T14:24:43.135 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/vg_nvme/lv_31 2024-03-16T14:24:43.162 INFO:tasks.nvme_loop:Connecting nvme_loop smithi026:/dev/vg_nvme/lv_4... 2024-03-16T14:24:43.162 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-16T14:24:43.241 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-03-16T14:24:43.278 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/vg_nvme/lv_41 2024-03-16T14:24:43.305 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:24:43.306 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T14:24:43.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo nvme list 2024-03-16T14:24:43.425 INFO:teuthology.orchestra.run.smithi026.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-16T14:24:43.425 INFO:teuthology.orchestra.run.smithi026.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-16T14:24:43.425 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/nvme0n1 PHMB74760020480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-16T14:24:43.426 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/nvme1n1 fce080e6f8ef851636ca Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:43.426 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/nvme2n1 0c6f6191e7343f2dbecc Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:43.426 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/nvme3n1 05203bf964821021fac4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:43.426 INFO:teuthology.orchestra.run.smithi026.stdout:/dev/nvme4n1 c0d34abe40ec614719cb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:43.427 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-16T14:24:43.427 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:24:43.427 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/scratch_devs 2024-03-16T14:24:43.492 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:24:43.492 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T14:24:43.511 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-16T14:24:43.511 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:24:40.327601943 +0000 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:17.473873441 +0000 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:17.473873441 +0000 2024-03-16T14:24:43.569 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:17.473873441 +0000 2024-03-16T14:24:43.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-16T14:24:43.639 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:24:43.639 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:24:43.639 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000228229 s, 2.2 MB/s 2024-03-16T14:24:43.641 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-16T14:24:43.699 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-03-16T14:24:43.756 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-16T14:24:43.756 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-16T14:24:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:43.756 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:24:40.327601943 +0000 2024-03-16T14:24:43.757 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:17.810869437 +0000 2024-03-16T14:24:43.757 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:17.810869437 +0000 2024-03-16T14:24:43.757 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:17.810869437 +0000 2024-03-16T14:24:43.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-16T14:24:43.824 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:24:43.824 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:24:43.824 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000199939 s, 2.6 MB/s 2024-03-16T14:24:43.826 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-16T14:24:43.883 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:24:40.327601943 +0000 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:18.148865422 +0000 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:18.148865422 +0000 2024-03-16T14:24:43.939 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:18.148865422 +0000 2024-03-16T14:24:43.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-16T14:24:44.006 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:24:44.006 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:24:44.006 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000188133 s, 2.7 MB/s 2024-03-16T14:24:44.008 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-16T14:24:44.064 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-03-16T14:24:44.120 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-16T14:24:44.120 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-16T14:24:44.120 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-16T14:24:44.120 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-16T14:24:44.120 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-03-16T14:24:44.121 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:24:40.327601943 +0000 2024-03-16T14:24:44.121 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:18.476861526 +0000 2024-03-16T14:24:44.121 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:18.476861526 +0000 2024-03-16T14:24:44.121 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:18.476861526 +0000 2024-03-16T14:24:44.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-16T14:24:44.186 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:24:44.186 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:24:44.186 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000255412 s, 2.0 MB/s 2024-03-16T14:24:44.187 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-16T14:24:44.247 DEBUG:teuthology.orchestra.run.smithi151:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-16T14:24:44.385 INFO:teuthology.orchestra.run.smithi151.stdout:loop 2024-03-16T14:24:44.386 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_1... 2024-03-16T14:24:44.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-16T14:24:44.424 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-16T14:24:44.461 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_11 2024-03-16T14:24:44.490 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_2... 2024-03-16T14:24:44.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-16T14:24:44.568 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-16T14:24:44.606 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_21 2024-03-16T14:24:44.660 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_3... 2024-03-16T14:24:44.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-16T14:24:44.736 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-16T14:24:44.775 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_31 2024-03-16T14:24:44.804 INFO:tasks.nvme_loop:Connecting nvme_loop smithi151:/dev/vg_nvme/lv_4... 2024-03-16T14:24:44.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-16T14:24:44.889 INFO:teuthology.orchestra.run.smithi151.stdout:1 2024-03-16T14:24:44.928 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/vg_nvme/lv_41 2024-03-16T14:24:44.965 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:24:44.965 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T14:24:45.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme list 2024-03-16T14:24:45.087 INFO:teuthology.orchestra.run.smithi151.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-16T14:24:45.087 INFO:teuthology.orchestra.run.smithi151.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-16T14:24:45.087 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme0n1 CVFT623300JJ400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-16T14:24:45.088 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme1n1 feac4ad875fb621cce0c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:45.088 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme2n1 7d915526d95c7031a872 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:45.088 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme3n1 142f0dd0ca95b7716786 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:45.088 INFO:teuthology.orchestra.run.smithi151.stdout:/dev/nvme4n1 6ad22d7cdd05cb024c38 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-16T14:24:45.090 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-16T14:24:45.090 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:24:45.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/scratch_devs 2024-03-16T14:24:45.157 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-16T14:24:45.248 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN'], 'sha1': '5ea4d9cfac4d9f2893126a48d28cca7da69243c1'} 2024-03-16T14:24:45.249 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 2024-03-16T14:24:45.249 INFO:tasks.cephadm:Cluster fsid is efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:24:45.250 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-16T14:24:45.250 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.26', 'mon.c': '[v2:172.21.15.26:3301,v1:172.21.15.26:6790]', 'mon.b': '172.21.15.151'} 2024-03-16T14:24:45.250 INFO:tasks.cephadm:First mon is mon.a on smithi026 2024-03-16T14:24:45.250 INFO:tasks.cephadm:First mgr is y 2024-03-16T14:24:45.251 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-16T14:24:45.251 DEBUG:teuthology.orchestra.run.smithi026:> sudo hostname $(hostname -s) 2024-03-16T14:24:45.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-03-16T14:24:45.311 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-16T14:24:45.311 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ea4d9cfac4d9f2893126a48d28cca7da69243c1 2024-03-16T14:24:45.529 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '5ea4d9cfac4d9f2893126a48d28cca7da69243c1', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77679/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2121-g5ea4d9cf', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2121.g5ea4d9cf'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-yuri6-testing-2024-03-15-0709/5ea4d9cfac4d9f2893126a48d28cca7da69243c1/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 16:56:55.955977', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri6-testing-2024-03-15-0709', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-yuri6-testing-2024-03-15-0709/5ea4d9cfac4d9f2893126a48d28cca7da69243c1/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-16T14:24:45.751 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-yuri6-testing-2024-03-15-0709, sha1 5ea4d9cfac4d9f2893126a48d28cca7da69243c1 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=5ea4d9cfac4d9f2893126a48d28cca7da69243c1 2024-03-16T14:24:45.752 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-yuri6-testing-2024-03-15-0709/5ea4d9cfac4d9f2893126a48d28cca7da69243c1/centos/9/x86_64/flavors/default/cephadm 2024-03-16T14:24:45.752 DEBUG:teuthology.orchestra.run.smithi026:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuri6-testing-2024-03-15-0709/5ea4d9cfac4d9f2893126a48d28cca7da69243c1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T14:24:46.354 INFO:teuthology.orchestra.run.smithi026.stdout:-rw-r--r--. 1 ubuntu ubuntu 767298 Mar 16 14:24 /home/ubuntu/cephtest/cephadm 2024-03-16T14:24:46.355 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-yuri6-testing-2024-03-15-0709/5ea4d9cfac4d9f2893126a48d28cca7da69243c1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-16T14:24:46.804 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r--. 1 ubuntu ubuntu 767298 Mar 16 14:24 /home/ubuntu/cephtest/cephadm 2024-03-16T14:24:46.804 DEBUG:teuthology.orchestra.run.smithi026:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T14:24:46.823 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-16T14:24:46.854 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 on all hosts... 2024-03-16T14:24:46.854 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 pull 2024-03-16T14:24:46.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 pull 2024-03-16T14:24:47.696 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1... 2024-03-16T14:24:47.788 INFO:teuthology.orchestra.run.smithi026.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1... 2024-03-16T14:25:14.513 INFO:teuthology.orchestra.run.smithi026.stdout:{ 2024-03-16T14:25:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "ceph_version": "ceph version 19.0.0-2121-g5ea4d9cf (5ea4d9cfac4d9f2893126a48d28cca7da69243c1) squid (dev)", 2024-03-16T14:25:14.513 INFO:teuthology.orchestra.run.smithi026.stdout: "image_id": "3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9", 2024-03-16T14:25:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "repo_digests": [ 2024-03-16T14:25:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348" 2024-03-16T14:25:14.514 INFO:teuthology.orchestra.run.smithi026.stdout: ] 2024-03-16T14:25:14.514 INFO:teuthology.orchestra.run.smithi026.stdout:} 2024-03-16T14:25:21.458 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-03-16T14:25:21.458 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.0.0-2121-g5ea4d9cf (5ea4d9cfac4d9f2893126a48d28cca7da69243c1) squid (dev)", 2024-03-16T14:25:21.458 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9", 2024-03-16T14:25:21.458 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-03-16T14:25:21.458 INFO:teuthology.orchestra.run.smithi151.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348" 2024-03-16T14:25:21.458 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-03-16T14:25:21.458 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-03-16T14:25:21.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph 2024-03-16T14:25:21.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-03-16T14:25:21.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /etc/ceph 2024-03-16T14:25:21.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-03-16T14:25:21.612 INFO:tasks.cephadm:Writing seed config... 2024-03-16T14:25:21.613 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-16T14:25:21.613 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-16T14:25:21.613 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-16T14:25:21.613 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-16T14:25:21.614 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-16T14:25:21.614 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-16T14:25:21.614 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-16T14:25:21.614 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-16T14:25:21.614 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-16T14:25:21.614 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-16T14:25:21.614 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:25:21.615 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-16T14:25:21.641 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = efcc9110-e3a0-11ee-95c9-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-16T14:25:21.641 DEBUG:teuthology.orchestra.run.smithi026:mon.a> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a.service 2024-03-16T14:25:21.684 DEBUG:teuthology.orchestra.run.smithi026:mgr.y> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y.service 2024-03-16T14:25:21.726 INFO:tasks.cephadm:Bootstrapping... 2024-03-16T14:25:21.726 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 -v bootstrap --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.26 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-16T14:25:21.923 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-16T14:25:21.923 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1', '-v', 'bootstrap', '--fsid', 'efcc9110-e3a0-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.26', '--skip-admin-label'] 2024-03-16T14:25:21.947 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T14:25:21.947 INFO:teuthology.orchestra.run.smithi026.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-16T14:25:21.947 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying podman|docker is present... 2024-03-16T14:25:21.970 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T14:25:21.970 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying lvm2 is present... 2024-03-16T14:25:21.970 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying time synchronization is in place... 2024-03-16T14:25:21.977 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T14:25:21.977 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T14:25:21.983 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T14:25:21.983 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-16T14:25:21.989 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-03-16T14:25:21.995 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-03-16T14:25:21.995 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-03-16T14:25:21.995 INFO:teuthology.orchestra.run.smithi026.stdout:Repeating the final host check... 2024-03-16T14:25:22.018 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-16T14:25:22.018 INFO:teuthology.orchestra.run.smithi026.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-16T14:25:22.018 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl is present 2024-03-16T14:25:22.018 INFO:teuthology.orchestra.run.smithi026.stdout:lvcreate is present 2024-03-16T14:25:22.024 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-16T14:25:22.024 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-16T14:25:22.030 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-16T14:25:22.030 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-16T14:25:22.036 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout enabled 2024-03-16T14:25:22.042 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout active 2024-03-16T14:25:22.042 INFO:teuthology.orchestra.run.smithi026.stdout:Unit chronyd.service is enabled and running 2024-03-16T14:25:22.042 INFO:teuthology.orchestra.run.smithi026.stdout:Host looks OK 2024-03-16T14:25:22.042 INFO:teuthology.orchestra.run.smithi026.stdout:Cluster fsid: efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:22.042 INFO:teuthology.orchestra.run.smithi026.stdout:Acquiring lock 140672216335744 on /run/cephadm/efcc9110-e3a0-11ee-95c9-87774f69a715.lock 2024-03-16T14:25:22.042 INFO:teuthology.orchestra.run.smithi026.stdout:Lock 140672216335744 acquired on /run/cephadm/efcc9110-e3a0-11ee-95c9-87774f69a715.lock 2024-03-16T14:25:22.043 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 3300 ... 2024-03-16T14:25:22.043 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying IP 172.21.15.26 port 6789 ... 2024-03-16T14:25:22.043 INFO:teuthology.orchestra.run.smithi026.stdout:Base mon IP(s) is [172.21.15.26:3300, 172.21.15.26:6789], mon addrv is [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-03-16T14:25:22.046 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.26 metric 100 2024-03-16T14:25:22.046 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.26 metric 100 2024-03-16T14:25:22.048 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-16T14:25:22.048 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-16T14:25:22.048 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-16T14:25:22.051 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-16T14:25:22.051 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-16T14:25:22.051 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T14:25:22.051 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-16T14:25:22.051 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:cac0/64 scope link noprefixroute 2024-03-16T14:25:22.051 INFO:teuthology.orchestra.run.smithi026.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-16T14:25:22.052 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-03-16T14:25:22.052 INFO:teuthology.orchestra.run.smithi026.stdout:Mon IP `172.21.15.26` is in CIDR network `172.21.0.0/20` 2024-03-16T14:25:22.052 INFO:teuthology.orchestra.run.smithi026.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-16T14:25:22.052 INFO:teuthology.orchestra.run.smithi026.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-16T14:25:22.053 INFO:teuthology.orchestra.run.smithi026.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1... 2024-03-16T14:25:22.548 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stdout 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 2024-03-16T14:25:22.548 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1... 2024-03-16T14:25:22.548 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Getting image source signatures 2024-03-16T14:25:22.548 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:9e5cf2925f3ee875bed70e02c03c261b6a1ea0e59712a525c907dbfe43e2f5e4 2024-03-16T14:25:22.548 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-16T14:25:22.548 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Copying config sha256:3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 2024-03-16T14:25:22.548 INFO:teuthology.orchestra.run.smithi026.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-16T14:25:23.436 INFO:teuthology.orchestra.run.smithi026.stdout:ceph: stdout ceph version 19.0.0-2121-g5ea4d9cf (5ea4d9cfac4d9f2893126a48d28cca7da69243c1) squid (dev) 2024-03-16T14:25:23.436 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph version: ceph version 19.0.0-2121-g5ea4d9cf (5ea4d9cfac4d9f2893126a48d28cca7da69243c1) squid (dev) 2024-03-16T14:25:23.436 INFO:teuthology.orchestra.run.smithi026.stdout:Extracting ceph user uid/gid from container image... 2024-03-16T14:25:24.441 INFO:teuthology.orchestra.run.smithi026.stdout:stat: stdout 167 167 2024-03-16T14:25:24.441 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial keys... 2024-03-16T14:25:25.238 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDUq/VlAY8aLRAAtoQedl0ZYjgdYzbnzToXMA== 2024-03-16T14:25:26.043 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDVq/VlUw/yHhAAWWQ8bjsk5X/MGMIUY4Q+5w== 2024-03-16T14:25:26.840 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph-authtool: stdout AQDWq/Vl8r/RExAA2Nn8hBpMkL/WJ+3TdzLsIA== 2024-03-16T14:25:26.841 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial monmap... 2024-03-16T14:25:27.737 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:monmaptool for a [v2:172.21.15.26:3300,v1:172.21.15.26:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:setting min_mon_release = quincy 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: set fsid to efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:25:27.738 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mon... 2024-03-16T14:25:28.560 INFO:teuthology.orchestra.run.smithi026.stdout:create mon.a on 2024-03-16T14:25:28.905 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-16T14:25:29.082 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715.target. 2024-03-16T14:25:29.082 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715.target. 2024-03-16T14:25:29.334 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a 2024-03-16T14:25:29.334 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-16T14:25:29.502 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715.target.wants/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@.service. 2024-03-16T14:25:30.017 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T14:25:30.017 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-16T14:25:30.027 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T14:25:30.027 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-16T14:25:30.027 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-16T14:25:30.027 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T14:25:30.027 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon to start... 2024-03-16T14:25:30.027 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mon... 2024-03-16T14:25:30.129 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:30 smithi026 ceph-mon[24837]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T14:25:30.996 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout cluster: 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout id: efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout services: 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.496938s) 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout data: 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-16T14:25:30.997 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-16T14:25:30.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout pgs: 2024-03-16T14:25:30.998 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:30.998 INFO:teuthology.orchestra.run.smithi026.stdout:mon is available 2024-03-16T14:25:30.998 INFO:teuthology.orchestra.run.smithi026.stdout:Assimilating anything we can from ceph.conf... 2024-03-16T14:25:31.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:31 smithi026 ceph-mon[24837]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T14:25:31.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:31 smithi026 ceph-mon[24837]: monmap e1: 1 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:25:31.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:31 smithi026 ceph-mon[24837]: fsmap 2024-03-16T14:25:31.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:31 smithi026 ceph-mon[24837]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T14:25:31.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:31 smithi026 ceph-mon[24837]: mgrmap e1: no daemons active 2024-03-16T14:25:31.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:31 smithi026 ceph-mon[24837]: from='client.? 172.21.15.26:0/4017791681' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-16T14:25:32.043 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:32.043 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T14:25:32.043 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.26:3300,v1:172.21.15.26:6789] 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T14:25:32.044 INFO:teuthology.orchestra.run.smithi026.stdout:Generating new minimal ceph.conf... 2024-03-16T14:25:32.262 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:32 smithi026 ceph-mon[24837]: from='client.? 172.21.15.26:0/1483592612' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T14:25:32.262 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:32 smithi026 ceph-mon[24837]: from='client.? 172.21.15.26:0/1483592612' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T14:25:33.057 INFO:teuthology.orchestra.run.smithi026.stdout:Restarting the monitor... 2024-03-16T14:25:33.308 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:33 smithi026 ceph-mon[24837]: from='client.? 172.21.15.26:0/335646503' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:25:33.308 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:33 smithi026 systemd[1]: Stopping Ceph mon.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:25:33.563 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:33 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a[24833]: 2024-03-16T14:25:33.303+0000 7f22bf871700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:25:33.563 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:33 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a[24833]: 2024-03-16T14:25:33.303+0000 7f22bf871700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-16T14:25:33.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:33 smithi026 podman[25094]: 2024-03-16 14:25:33.560700682 +0000 UTC m=+0.335095637 container died c5ed3595c1ab0a91fd2284cb1c1e06ad53a70ffa66b182dbfbb265b810c67f98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-16T14:25:33.840 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:33 smithi026 podman[25094]: 2024-03-16 14:25:33.837025172 +0000 UTC m=+0.611420127 container cleanup c5ed3595c1ab0a91fd2284cb1c1e06ad53a70ffa66b182dbfbb265b810c67f98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-16T14:25:34.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:33 smithi026 bash[25094]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a 2024-03-16T14:25:36.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:35 smithi026 podman[25108]: 2024-03-16 14:25:35.6825374 +0000 UTC m=+2.121302503 container remove c5ed3595c1ab0a91fd2284cb1c1e06ad53a70ffa66b182dbfbb265b810c67f98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c) 2024-03-16T14:25:36.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:35 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-16T14:25:36.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:35 smithi026 systemd[1]: Stopped Ceph mon.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:25:36.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:35 smithi026 systemd[1]: Starting Ceph mon.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:25:36.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:36 smithi026 podman[25200]: 2024-03-16 14:25:35.981035925 +0000 UTC m=+0.017585284 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 2024-03-16T14:25:37.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:37 smithi026 podman[25200]: 2024-03-16 14:25:37.267684736 +0000 UTC m=+1.304234124 container create 3b7c51472ad641330f72f717db07ee1433f2e67b0ad73815d12547dcdd6db928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T14:25:39.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 podman[25200]: 2024-03-16 14:25:39.141212135 +0000 UTC m=+3.177761481 container init 3b7c51472ad641330f72f717db07ee1433f2e67b0ad73815d12547dcdd6db928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T14:25:39.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 podman[25200]: 2024-03-16 14:25:39.146638198 +0000 UTC m=+3.183187545 container start 3b7c51472ad641330f72f717db07ee1433f2e67b0ad73815d12547dcdd6db928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:25:39.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: ceph version 19.0.0-2121-g5ea4d9cf (5ea4d9cfac4d9f2893126a48d28cca7da69243c1) squid (dev), process ceph-mon, pid 2 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: pidfile_write: ignore empty --pid-file 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: load: jerasure load: lrc 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: RocksDB version: 7.9.2 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Git sha 0 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Compile date 2024-03-15 15:33:47 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: DB SUMMARY 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: DB Session ID: CAC5YB9C51PVXW1GVZWX 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: CURRENT file: CURRENT 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T14:25:39.508 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84959 ; 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.error_if_exists: 0 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.create_if_missing: 0 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.paranoid_checks: 1 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.env: 0x55566962cb20 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T14:25:39.509 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.info_log: 0x55566a789920 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.statistics: (nil) 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.use_fsync: 0 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_log_file_size: 0 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.allow_fallocate: 1 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.use_direct_reads: 0 2024-03-16T14:25:39.510 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.db_log_dir: 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.wal_dir: 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.write_buffer_manager: 0x55566a75c1e0 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T14:25:39.511 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.unordered_write: 0 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.row_cache: None 2024-03-16T14:25:39.512 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.wal_filter: None 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.two_write_queues: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.wal_compression: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.atomic_flush: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T14:25:39.514 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.log_readahead_size: 0 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_background_jobs: 2 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_background_compactions: -1 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_subcompactions: 1 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T14:25:39.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_open_files: -1 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_background_flushes: -1 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Compression algorithms supported: 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kZSTD supported: 0 2024-03-16T14:25:39.516 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kXpressCompression supported: 0 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kZlibCompression supported: 1 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kSnappyCompression supported: 1 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kLZ4Compression supported: 1 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: kBZip2Compression supported: 0 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.merge_operator: 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_filter: None 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T14:25:39.517 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55566a789a20) 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: cache_index_and_filter_blocks: 1 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: pin_top_level_index_and_filter: 1 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_type: 0 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: data_block_index_type: 0 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_shortening: 1 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: checksum: 4 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: no_block_cache: 0 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache: 0x55566a78f090 2024-03-16T14:25:39.518 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_name: BinnedLRUCache 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_options: 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: capacity : 536870912 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: num_shard_bits : 4 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: strict_capacity_limit : 0 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: high_pri_pool_ratio: 0.000 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_cache_compressed: (nil) 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: persistent_cache: (nil) 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_size: 4096 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_size_deviation: 10 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_restart_interval: 16 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: index_block_restart_interval: 1 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: metadata_block_size: 4096 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: partition_filters: 0 2024-03-16T14:25:39.519 INFO:journalctl@ceph.mon.a.smithi026.stdout: use_delta_encoding: 1 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: filter_policy: bloomfilter 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: whole_key_filtering: 1 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: verify_compression: 0 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: read_amp_bytes_per_bit: 0 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: format_version: 5 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: enable_index_compression: 1 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: block_align: 0 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: max_auto_readahead_size: 262144 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: prepopulate_block_cache: 0 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: initial_auto_readahead_size: 8192 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T14:25:39.520 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression: NoCompression 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.num_levels: 7 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T14:25:39.521 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T14:25:39.522 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T14:25:39.523 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T14:25:39.524 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.table_properties_collectors: 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.inplace_update_support: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.bloom_locality: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.max_successive_merges: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T14:25:39.525 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.ttl: 2592000 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.enable_blob_files: false 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.min_blob_size: 0 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T14:25:39.526 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 167b5dba-af08-4227-a540-f300a68a7e71 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710599139178154, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710599139179124, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81584, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79771, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 74080, "raw_average_value_size": 351, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710599139, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "167b5dba-af08-4227-a540-f300a68a7e71", "db_session_id": "CAC5YB9C51PVXW1GVZWX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710599139179221, "job": 1, "event": "recovery_finished"} 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T14:25:39.527 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55566a882000 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: DB pointer 0x55566a86c000 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** DB Stats ** 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T14:25:39.528 INFO:journalctl@ceph.mon.a.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: L0 2/0 81.51 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: Sum 2/0 81.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 120.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T14:25:39.529 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: Cumulative compaction: 0.00 GB write, 23.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: Interval compaction: 0.00 GB write, 23.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: Block cache BinnedLRUCache@0x55566a78f090#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T14:25:39.530 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: starting mon.a rank 0 at public addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] at bind addrs [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???) e1 preinit fsid efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???).mds e1 new map 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???).mds e1 print_map 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout: e1 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout: legacy client fscid: -1 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout: 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout: No filesystems configured 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-16T14:25:39.531 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-16T14:25:39.532 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: monmap e1: 1 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:25:39.532 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: fsmap 2024-03-16T14:25:39.532 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: osdmap e1: 0 total, 0 up, 0 in 2024-03-16T14:25:39.532 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:39 smithi026 ceph-mon[25218]: mgrmap e1: no daemons active 2024-03-16T14:25:40.043 INFO:teuthology.orchestra.run.smithi026.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-16T14:25:40.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:40 smithi026 bash[25200]: 3b7c51472ad641330f72f717db07ee1433f2e67b0ad73815d12547dcdd6db928 2024-03-16T14:25:40.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:40 smithi026 systemd[1]: Started Ceph mon.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:25:44.465 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:44 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2197398936' entity='client.admin' 2024-03-16T14:25:44.466 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-16T14:25:44.487 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-16T14:25:44.487 INFO:teuthology.orchestra.run.smithi026.stdout:Creating mgr... 2024-03-16T14:25:44.488 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-16T14:25:44.488 INFO:teuthology.orchestra.run.smithi026.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-16T14:25:44.750 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y 2024-03-16T14:25:44.750 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Failed to reset failed state of unit ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y.service: Unit ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y.service not loaded. 2024-03-16T14:25:44.905 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715.target.wants/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@.service. 2024-03-16T14:25:45.106 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:44 smithi026 systemd[1]: Starting Ceph mgr.y for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:25:45.375 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 podman[25435]: 2024-03-16 14:25:45.103376508 +0000 UTC m=+0.084286119 container create cfd474663860976b1402d5f59900e857957aa5fbe47f14f0a43396bfe538e8dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307) 2024-03-16T14:25:45.375 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 podman[25435]: 2024-03-16 14:25:45.037275417 +0000 UTC m=+0.018185022 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 2024-03-16T14:25:45.376 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 podman[25435]: 2024-03-16 14:25:45.280467483 +0000 UTC m=+0.261377090 container init cfd474663860976b1402d5f59900e857957aa5fbe47f14f0a43396bfe538e8dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-16T14:25:45.376 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 podman[25435]: 2024-03-16 14:25:45.282955508 +0000 UTC m=+0.263865112 container start cfd474663860976b1402d5f59900e857957aa5fbe47f14f0a43396bfe538e8dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T14:25:45.376 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 bash[25435]: cfd474663860976b1402d5f59900e857957aa5fbe47f14f0a43396bfe538e8dc 2024-03-16T14:25:45.388 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T14:25:45.388 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-16T14:25:45.395 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T14:25:45.396 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-16T14:25:45.396 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-16T14:25:45.396 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to enable service . firewalld.service is not available 2024-03-16T14:25:45.404 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T14:25:45.404 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-16T14:25:45.411 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T14:25:45.411 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-16T14:25:45.411 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-16T14:25:45.411 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-16T14:25:45.411 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr to start... 2024-03-16T14:25:45.411 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr... 2024-03-16T14:25:45.647 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 systemd[1]: Started Ceph mgr.y for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:25:45.910 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:45.848+0000 7f81ea086200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T14:25:46.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:45 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4210705297' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T14:25:46.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:45.947+0000 7f81ea086200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T14:25:46.463 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "efcc9110-e3a0-11ee-95c9-87774f69a715", 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:46.464 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T14:25:46.465 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T14:25:46.466 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T14:25:30.003474+0000", 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-16T14:25:46.467 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (1/15)... 2024-03-16T14:25:46.952 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:46 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:46.612+0000 7f81ea086200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T14:25:46.952 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:46 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:46.707+0000 7f81ea086200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T14:25:46.952 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:46 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:46.850+0000 7f81ea086200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T14:25:46.952 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:46 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:46.947+0000 7f81ea086200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T14:25:47.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:47 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:47.031+0000 7f81ea086200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T14:25:47.598 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:47 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:47.299+0000 7f81ea086200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T14:25:47.599 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:47 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:47.411+0000 7f81ea086200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T14:25:47.599 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:47 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:47.498+0000 7f81ea086200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T14:25:47.599 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:47 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:47.594+0000 7f81ea086200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T14:25:48.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:47 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:47.771+0000 7f81ea086200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T14:25:48.952 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:48 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:48.771+0000 7f81ea086200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T14:25:48.952 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:48 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:48.904+0000 7f81ea086200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T14:25:49.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:49 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/726505414' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T14:25:49.506 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:49.245+0000 7f81ea086200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T14:25:49.506 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:49.344+0000 7f81ea086200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T14:25:50.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:49.542+0000 7f81ea086200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:25:50.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:49.729+0000 7f81ea086200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T14:25:50.282 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:50 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:50.016+0000 7f81ea086200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:25:50.283 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:50 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:50.278+0000 7f81ea086200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T14:25:50.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:50 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:50.520+0000 7f81ea086200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T14:25:51.345 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:51.345 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-16T14:25:51.345 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "efcc9110-e3a0-11ee-95c9-87774f69a715", 2024-03-16T14:25:51.345 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-16T14:25:51.346 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T14:25:51.347 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T14:25:51.348 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:51.349 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T14:25:51.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:51.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T14:25:30.003474+0000", 2024-03-16T14:25:51.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T14:25:51.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:51.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T14:25:51.350 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-16T14:25:51.350 INFO:teuthology.orchestra.run.smithi026.stdout:mgr not available, waiting (2/15)... 2024-03-16T14:25:51.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:51 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:51.369+0000 7f81ea086200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T14:25:52.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:51 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:51.949+0000 7f81ea086200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T14:25:52.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:52.048+0000 7f81ea086200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T14:25:52.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:52.267+0000 7f81ea086200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: Activating manager daemon y 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: mgrmap e2: y(active, starting, since 0.00290128s) 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: Manager daemon y is now available 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:25:52.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' 2024-03-16T14:25:52.758 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:25:52.758 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' 2024-03-16T14:25:52.758 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:52 smithi026 ceph-mon[25218]: from='mgr.14100 172.21.15.26:0/1999296030' entity='mgr.y' 2024-03-16T14:25:54.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:54 smithi026 ceph-mon[25218]: mgrmap e3: y(active, since 1.00699s) 2024-03-16T14:25:54.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:54 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/744540828' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-16T14:25:54.659 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:54.659 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-16T14:25:54.659 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsid": "efcc9110-e3a0-11ee-95c9-87774f69a715", 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "health": { 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 0 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "a" 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-16T14:25:54.660 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-16T14:25:54.661 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-16T14:25:54.662 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "restful" 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ], 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "modified": "2024-03-16T14:25:30.003474+0000", 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout }, 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-16T14:25:54.663 INFO:teuthology.orchestra.run.smithi026.stdout:mgr is available 2024-03-16T14:25:55.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:55 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/596976991' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-16T14:25:55.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:55 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/596976991' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-16T14:25:55.760 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [global] 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout fsid = efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout [osd] 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-16T14:25:55.761 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-16T14:25:55.762 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling cephadm module... 2024-03-16T14:25:56.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:56 smithi026 ceph-mon[25218]: mgrmap e4: y(active, since 3s) 2024-03-16T14:25:57.595 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:57 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3983039836' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-16T14:25:57.596 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:57 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ignoring --setuser ceph since I am not root 2024-03-16T14:25:57.596 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:57 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ignoring --setgroup ceph since I am not root 2024-03-16T14:25:58.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:57 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:57.870+0000 7fc16bfe2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T14:25:58.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:57 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:57.969+0000 7fc16bfe2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T14:25:58.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:58 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3983039836' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-16T14:25:58.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:58 smithi026 ceph-mon[25218]: mgrmap e5: y(active, since 5s) 2024-03-16T14:25:58.876 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:58 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:58.632+0000 7fc16bfe2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T14:25:58.876 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:58 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:58.729+0000 7fc16bfe2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T14:25:58.876 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:58 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:58.872+0000 7fc16bfe2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T14:25:59.193 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:58 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:58.974+0000 7fc16bfe2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T14:25:59.193 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:59 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:59.059+0000 7fc16bfe2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T14:25:59.444 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:25:59 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1259928985' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T14:25:59.444 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:59 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:59.329+0000 7fc16bfe2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T14:25:59.444 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:59 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:59.440+0000 7fc16bfe2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-03-16T14:25:59.506 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 5... 2024-03-16T14:25:59.736 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:59 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:59.532+0000 7fc16bfe2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T14:25:59.736 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:59 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:59.630+0000 7fc16bfe2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T14:26:00.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:25:59 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:25:59.809+0000 7fc16bfe2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T14:26:01.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:00 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:00.793+0000 7fc16bfe2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T14:26:01.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:00 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:00.926+0000 7fc16bfe2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T14:26:01.555 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:01 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:01.262+0000 7fc16bfe2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T14:26:01.555 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:01 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:01.353+0000 7fc16bfe2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T14:26:01.555 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:01 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:01.552+0000 7fc16bfe2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:26:02.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:01 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:01.739+0000 7fc16bfe2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T14:26:02.292 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:02 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:02.028+0000 7fc16bfe2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:26:02.292 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:02 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:02.288+0000 7fc16bfe2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T14:26:02.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:02 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:02.527+0000 7fc16bfe2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T14:26:03.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:03 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:03.346+0000 7fc16bfe2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T14:26:04.246 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:03 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:03.924+0000 7fc16bfe2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T14:26:04.246 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:04.023+0000 7fc16bfe2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T14:26:04.246 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:04.243+0000 7fc16bfe2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T14:26:04.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: Active manager daemon y restarted 2024-03-16T14:26:04.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: Activating manager daemon y 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: osdmap e2: 0 total, 0 up, 0 in 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: mgrmap e6: y(active, starting, since 0.00297172s) 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: Manager daemon y is now available 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:04.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:04 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:05.549 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:05 smithi026 ceph-mon[25218]: Found migration_current of "None". Setting to last migration. 2024-03-16T14:26:05.549 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:05 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:26:05.549 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:05 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:26:05.549 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:05 smithi026 ceph-mon[25218]: mgrmap e7: y(active, since 1.00526s) 2024-03-16T14:26:05.549 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:05 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:05.550 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:05 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:05.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-16T14:26:05.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-16T14:26:05.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T14:26:05.786 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-16T14:26:05.786 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 5 is available 2024-03-16T14:26:05.786 INFO:teuthology.orchestra.run.smithi026.stdout:Setting orchestrator backend to cephadm... 2024-03-16T14:26:06.437 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:06 smithi026 ceph-mon[25218]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T14:26:06.437 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:06 smithi026 ceph-mon[25218]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T14:26:06.437 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:06 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:07.357 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:07 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:05] ENGINE Bus STARTING 2024-03-16T14:26:07.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:07 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:05] ENGINE Serving on https://172.21.15.26:7150 2024-03-16T14:26:07.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:07 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:06] ENGINE Serving on http://172.21.15.26:8765 2024-03-16T14:26:07.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:07 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:06] ENGINE Bus STARTED 2024-03-16T14:26:07.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:07 smithi026 ceph-mon[25218]: mgrmap e8: y(active, since 2s) 2024-03-16T14:26:07.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:07 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:07.358 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:07 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:08.140 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-16T14:26:08.140 INFO:teuthology.orchestra.run.smithi026.stdout:Generating ssh key... 2024-03-16T14:26:08.498 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-mon[25218]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: Generating public/private rsa key pair. 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: Your identification has been saved in /tmp/tmpmwdljelj/key. 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: Your public key has been saved in /tmp/tmpmwdljelj/key.pub. 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: The key fingerprint is: 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: SHA256:xXHLZD2VNneWvuykVFNC+By4z4+gllddqvpXoNaCVhI ceph-efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: The key's randomart image is: 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: +---[RSA 3072]----+ 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | . +=..+| 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | .E*o.===| 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | o.o+o==| 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | .. o ++.| 2024-03-16T14:26:09.257 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | S + =o++| 2024-03-16T14:26:09.258 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | o =.*+o| 2024-03-16T14:26:09.258 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | . +.=++ | 2024-03-16T14:26:09.258 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | + o.o..| 2024-03-16T14:26:09.258 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: | ..+.. | 2024-03-16T14:26:09.258 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: +----[SHA256]-----+ 2024-03-16T14:26:09.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:09 smithi026 ceph-mon[25218]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:09.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:09 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:09.669 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:09 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:10.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:10 smithi026 ceph-mon[25218]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:10.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:10 smithi026 ceph-mon[25218]: Generating ssh key... 2024-03-16T14:26:10.785 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfvpuP1/ONEzg0jtWz65rIiYpnptaRu8GtfRwNe+vC2hOST93VOPDHm+5PGAP0qDlb6HZ90ASd/iCcwGJ2yoF5tKL1JbsxCmENjiCUabwMoZ5i387zqE0qWll5PY1tq+2BqTzRTpkDoWPusd3LFIWfISnjAbIBY7ARwUlIc4fLoksL4T6lLQZsnOG3pCFiqCRqrEdMATHWI6+lBPK70igqwJD6E/wU8zT19a/Qe90dszocskK5iUcj1w48WbPgj7SypJf5126doP6h4iQQx4JvvvzhhSQFsfoVuCktJtHNQ8xm72w1gTxtE3Tm0Q45LumxxUmcApFKZP9Qo0t87QdQtMZ3KS4Q8ktmR2ZehZWxSP0UiR1nJYK/eq5uxNJxiHlkrOtrg0ylWxSHyyXLHhhZDYw8RUu96mpSuKjTD822nbb30SRTD8aqiXL7/3BNqd58XMVOFg20PfzecxI6Mrv9aSOLq7aPVptCfVeQ2/A7SqSs2WZu4jgIFuifEzaMBK8= ceph-efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:26:10.786 INFO:teuthology.orchestra.run.smithi026.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-16T14:26:10.786 INFO:teuthology.orchestra.run.smithi026.stdout:Adding key to root@localhost authorized_keys... 2024-03-16T14:26:10.786 INFO:teuthology.orchestra.run.smithi026.stdout:Adding host smithi026... 2024-03-16T14:26:11.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:11 smithi026 ceph-mon[25218]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:12.575 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:12 smithi026 ceph-mon[25218]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi026", "addr": "172.21.15.26", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:12.576 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:12 smithi026 ceph-mon[25218]: Deploying cephadm binary to smithi026 2024-03-16T14:26:16.233 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Added host 'smithi026' with addr '172.21.15.26' 2024-03-16T14:26:16.234 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying unmanaged mon service... 2024-03-16T14:26:17.003 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:16 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:17.003 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:16 smithi026 ceph-mon[25218]: Added host smithi026 2024-03-16T14:26:17.003 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:16 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-16T14:26:17.669 INFO:teuthology.orchestra.run.smithi026.stdout:Deploying unmanaged mgr service... 2024-03-16T14:26:18.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:17 smithi026 ceph-mon[25218]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:18.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:17 smithi026 ceph-mon[25218]: Saving service mon spec with placement count:5 2024-03-16T14:26:18.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:17 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:19.362 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-16T14:26:20.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:19 smithi026 ceph-mon[25218]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:20.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:19 smithi026 ceph-mon[25218]: Saving service mgr spec with placement count:2 2024-03-16T14:26:20.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:19 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:20.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:19 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:20.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:19 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:21.337 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:21 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1990326795' entity='client.admin' 2024-03-16T14:26:22.411 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling the dashboard module... 2024-03-16T14:26:22.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:22 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:22.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:22 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1425016599' entity='client.admin' 2024-03-16T14:26:23.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:23 smithi026 ceph-mon[25218]: from='mgr.14118 172.21.15.26:0/2252166820' entity='mgr.y' 2024-03-16T14:26:23.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:23 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/273513099' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-16T14:26:23.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:23 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ignoring --setuser ceph since I am not root 2024-03-16T14:26:23.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:23 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ignoring --setgroup ceph since I am not root 2024-03-16T14:26:24.177 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:24 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:24.078+0000 7fbc77343200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T14:26:24.177 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:24 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:24.173+0000 7fbc77343200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T14:26:24.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:24 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/273513099' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-16T14:26:24.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:24 smithi026 ceph-mon[25218]: mgrmap e9: y(active, since 19s) 2024-03-16T14:26:25.088 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:24 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:24.841+0000 7fbc77343200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T14:26:25.089 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:24 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:24.939+0000 7fbc77343200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T14:26:25.089 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:25.085+0000 7fbc77343200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T14:26:25.501 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:25.183+0000 7fbc77343200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T14:26:25.501 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:25.272+0000 7fbc77343200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T14:26:25.752 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2767348323' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-16T14:26:25.753 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:25.544+0000 7fbc77343200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T14:26:25.753 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:25.656+0000 7fbc77343200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T14:26:25.753 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:25.744+0000 7fbc77343200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T14:26:25.755 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-16T14:26:25.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-16T14:26:25.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-16T14:26:25.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-16T14:26:25.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-16T14:26:25.756 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-16T14:26:25.756 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for the mgr to restart... 2024-03-16T14:26:25.756 INFO:teuthology.orchestra.run.smithi026.stdout:Waiting for mgr epoch 9... 2024-03-16T14:26:26.004 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:25 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:25.851+0000 7fbc77343200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T14:26:26.506 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:26 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:26.032+0000 7fbc77343200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T14:26:27.498 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:27.022+0000 7fbc77343200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T14:26:27.498 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:27.156+0000 7fbc77343200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T14:26:27.498 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:27.494+0000 7fbc77343200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T14:26:27.756 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:27.586+0000 7fbc77343200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T14:26:28.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:27.787+0000 7fbc77343200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:26:28.256 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:27.978+0000 7fbc77343200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T14:26:28.535 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:28 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:28.269+0000 7fbc77343200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:26:28.535 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:28 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:28.531+0000 7fbc77343200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T14:26:29.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:28 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:28.774+0000 7fbc77343200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T14:26:30.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:29 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:29.603+0000 7fbc77343200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T14:26:30.506 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:30.186+0000 7fbc77343200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T14:26:30.506 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:30.289+0000 7fbc77343200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T14:26:31.006 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:26:30.514+0000 7fbc77343200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T14:26:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: Active manager daemon y restarted 2024-03-16T14:26:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: Activating manager daemon y 2024-03-16T14:26:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: osdmap e3: 0 total, 0 up, 0 in 2024-03-16T14:26:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: mgrmap e10: y(active, starting, since 0.0034278s) 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: Manager daemon y is now available 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:31.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:31.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:26:31.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:26:31.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:31 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:31] ENGINE Bus STARTING 2024-03-16T14:26:31.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:31 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:31] ENGINE Serving on https://172.21.15.26:7150 2024-03-16T14:26:31.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:31 smithi026 ceph-mon[25218]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-16T14:26:31.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:31 smithi026 ceph-mon[25218]: mgrmap e11: y(active, since 1.0056s) 2024-03-16T14:26:31.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:31 smithi026 ceph-mon[25218]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-16T14:26:32.430 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout { 2024-03-16T14:26:32.433 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-16T14:26:32.433 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-16T14:26:32.433 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout } 2024-03-16T14:26:32.433 INFO:teuthology.orchestra.run.smithi026.stdout:mgr epoch 9 is available 2024-03-16T14:26:32.433 INFO:teuthology.orchestra.run.smithi026.stdout:Generating a dashboard self-signed certificate... 2024-03-16T14:26:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:33 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:31] ENGINE Serving on http://172.21.15.26:8765 2024-03-16T14:26:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:33 smithi026 ceph-mon[25218]: [16/Mar/2024:14:26:31] ENGINE Bus STARTED 2024-03-16T14:26:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:33.719 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-16T14:26:33.719 INFO:teuthology.orchestra.run.smithi026.stdout:Creating initial admin user... 2024-03-16T14:26:34.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:34 smithi026 ceph-mon[25218]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:34.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:34 smithi026 ceph-mon[25218]: mgrmap e12: y(active, since 2s) 2024-03-16T14:26:34.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:34 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:34.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:34 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:34.376 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:34 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:35.254 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$P38tJszxuaYwUVXao7cz3OYQsllAS2P0t0xPi5zukQ4PNLpMSPrHq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710599194, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-16T14:26:35.254 INFO:teuthology.orchestra.run.smithi026.stdout:Fetching dashboard port number... 2024-03-16T14:26:35.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:35 smithi026 ceph-mon[25218]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:35.360 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:35 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:36.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:36 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1098547146' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-16T14:26:36.521 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stdout 8443 2024-03-16T14:26:36.531 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-16T14:26:36.531 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout disabled 2024-03-16T14:26:36.539 INFO:teuthology.orchestra.run.smithi026.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-16T14:26:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:systemctl: stdout inactive 2024-03-16T14:26:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:firewalld.service is not enabled 2024-03-16T14:26:36.540 INFO:teuthology.orchestra.run.smithi026.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-16T14:26:36.541 INFO:teuthology.orchestra.run.smithi026.stdout:Ceph Dashboard is now available at: 2024-03-16T14:26:36.541 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:36.541 INFO:teuthology.orchestra.run.smithi026.stdout: URL: https://smithi026.front.sepia.ceph.com:8443/ 2024-03-16T14:26:36.542 INFO:teuthology.orchestra.run.smithi026.stdout: User: admin 2024-03-16T14:26:36.542 INFO:teuthology.orchestra.run.smithi026.stdout: Password: jbhehs1hsq 2024-03-16T14:26:36.542 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:36.542 INFO:teuthology.orchestra.run.smithi026.stdout:Saving cluster configuration to /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config directory 2024-03-16T14:26:36.542 INFO:teuthology.orchestra.run.smithi026.stdout:Enabling autotune for osd_memory_target 2024-03-16T14:26:38.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:38 smithi026 ceph-mon[25218]: mgrmap e13: y(active, since 6s) 2024-03-16T14:26:39.024 INFO:teuthology.orchestra.run.smithi026.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-16T14:26:39.024 INFO:teuthology.orchestra.run.smithi026.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout:Or, if you are only running a single cluster on this host: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: ceph telemetry on 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.025 INFO:teuthology.orchestra.run.smithi026.stdout:For more information see: 2024-03-16T14:26:39.026 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.026 INFO:teuthology.orchestra.run.smithi026.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-16T14:26:39.026 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:39.026 INFO:teuthology.orchestra.run.smithi026.stdout:Bootstrap complete. 2024-03-16T14:26:39.071 INFO:tasks.cephadm:Fetching config... 2024-03-16T14:26:39.071 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:26:39.071 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-16T14:26:39.087 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-16T14:26:39.088 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:26:39.088 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-16T14:26:39.144 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-16T14:26:39.145 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:26:39.145 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-16T14:26:39.212 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-16T14:26:39.213 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:26:39.213 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-16T14:26:39.268 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-16T14:26:39.268 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfvpuP1/ONEzg0jtWz65rIiYpnptaRu8GtfRwNe+vC2hOST93VOPDHm+5PGAP0qDlb6HZ90ASd/iCcwGJ2yoF5tKL1JbsxCmENjiCUabwMoZ5i387zqE0qWll5PY1tq+2BqTzRTpkDoWPusd3LFIWfISnjAbIBY7ARwUlIc4fLoksL4T6lLQZsnOG3pCFiqCRqrEdMATHWI6+lBPK70igqwJD6E/wU8zT19a/Qe90dszocskK5iUcj1w48WbPgj7SypJf5126doP6h4iQQx4JvvvzhhSQFsfoVuCktJtHNQ8xm72w1gTxtE3Tm0Q45LumxxUmcApFKZP9Qo0t87QdQtMZ3KS4Q8ktmR2ZehZWxSP0UiR1nJYK/eq5uxNJxiHlkrOtrg0ylWxSHyyXLHhhZDYw8RUu96mpSuKjTD822nbb30SRTD8aqiXL7/3BNqd58XMVOFg20PfzecxI6Mrv9aSOLq7aPVptCfVeQ2/A7SqSs2WZu4jgIFuifEzaMBK8= ceph-efcc9110-e3a0-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T14:26:39.348 INFO:teuthology.orchestra.run.smithi026.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfvpuP1/ONEzg0jtWz65rIiYpnptaRu8GtfRwNe+vC2hOST93VOPDHm+5PGAP0qDlb6HZ90ASd/iCcwGJ2yoF5tKL1JbsxCmENjiCUabwMoZ5i387zqE0qWll5PY1tq+2BqTzRTpkDoWPusd3LFIWfISnjAbIBY7ARwUlIc4fLoksL4T6lLQZsnOG3pCFiqCRqrEdMATHWI6+lBPK70igqwJD6E/wU8zT19a/Qe90dszocskK5iUcj1w48WbPgj7SypJf5126doP6h4iQQx4JvvvzhhSQFsfoVuCktJtHNQ8xm72w1gTxtE3Tm0Q45LumxxUmcApFKZP9Qo0t87QdQtMZ3KS4Q8ktmR2ZehZWxSP0UiR1nJYK/eq5uxNJxiHlkrOtrg0ylWxSHyyXLHhhZDYw8RUu96mpSuKjTD822nbb30SRTD8aqiXL7/3BNqd58XMVOFg20PfzecxI6Mrv9aSOLq7aPVptCfVeQ2/A7SqSs2WZu4jgIFuifEzaMBK8= ceph-efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:26:39.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfvpuP1/ONEzg0jtWz65rIiYpnptaRu8GtfRwNe+vC2hOST93VOPDHm+5PGAP0qDlb6HZ90ASd/iCcwGJ2yoF5tKL1JbsxCmENjiCUabwMoZ5i387zqE0qWll5PY1tq+2BqTzRTpkDoWPusd3LFIWfISnjAbIBY7ARwUlIc4fLoksL4T6lLQZsnOG3pCFiqCRqrEdMATHWI6+lBPK70igqwJD6E/wU8zT19a/Qe90dszocskK5iUcj1w48WbPgj7SypJf5126doP6h4iQQx4JvvvzhhSQFsfoVuCktJtHNQ8xm72w1gTxtE3Tm0Q45LumxxUmcApFKZP9Qo0t87QdQtMZ3KS4Q8ktmR2ZehZWxSP0UiR1nJYK/eq5uxNJxiHlkrOtrg0ylWxSHyyXLHhhZDYw8RUu96mpSuKjTD822nbb30SRTD8aqiXL7/3BNqd58XMVOFg20PfzecxI6Mrv9aSOLq7aPVptCfVeQ2/A7SqSs2WZu4jgIFuifEzaMBK8= ceph-efcc9110-e3a0-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-16T14:26:39.410 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfvpuP1/ONEzg0jtWz65rIiYpnptaRu8GtfRwNe+vC2hOST93VOPDHm+5PGAP0qDlb6HZ90ASd/iCcwGJ2yoF5tKL1JbsxCmENjiCUabwMoZ5i387zqE0qWll5PY1tq+2BqTzRTpkDoWPusd3LFIWfISnjAbIBY7ARwUlIc4fLoksL4T6lLQZsnOG3pCFiqCRqrEdMATHWI6+lBPK70igqwJD6E/wU8zT19a/Qe90dszocskK5iUcj1w48WbPgj7SypJf5126doP6h4iQQx4JvvvzhhSQFsfoVuCktJtHNQ8xm72w1gTxtE3Tm0Q45LumxxUmcApFKZP9Qo0t87QdQtMZ3KS4Q8ktmR2ZehZWxSP0UiR1nJYK/eq5uxNJxiHlkrOtrg0ylWxSHyyXLHhhZDYw8RUu96mpSuKjTD822nbb30SRTD8aqiXL7/3BNqd58XMVOFg20PfzecxI6Mrv9aSOLq7aPVptCfVeQ2/A7SqSs2WZu4jgIFuifEzaMBK8= ceph-efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:26:39.426 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-16T14:26:39.668 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:26:39.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:39 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1496335747' entity='client.admin' 2024-03-16T14:26:42.804 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-16T14:26:42.804 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-16T14:26:43.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:42 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3978092826' entity='client.admin' 2024-03-16T14:26:43.309 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:26:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi026", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:26:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:45.907 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi151 2024-03-16T14:26:45.908 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:26:45.908 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.conf 2024-03-16T14:26:45.926 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:26:45.927 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:26:45.984 INFO:tasks.cephadm:Adding host smithi151 to orchestrator... 2024-03-16T14:26:45.984 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch host add smithi151 2024-03-16T14:26:46.222 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:26:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:46.472 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:26:46.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:26:46.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-16T14:26:46.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:26:46.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:46.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:46.473 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:47.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:47 smithi026 ceph-mon[25218]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:26:47.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:47 smithi026 ceph-mon[25218]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:26:49.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:48 smithi026 ceph-mon[25218]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:26:50.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:49 smithi026 ceph-mon[25218]: Deploying cephadm binary to smithi151 2024-03-16T14:26:52.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:51 smithi026 ceph-mon[25218]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:26:52.529 INFO:teuthology.orchestra.run.smithi026.stdout:Added host 'smithi151' with addr '172.21.15.151' 2024-03-16T14:26:53.119 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-16T14:26:53.356 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:26:53.743 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:53 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:53.743 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:53 smithi026 ceph-mon[25218]: Added host smithi151 2024-03-16T14:26:53.743 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:53 smithi026 ceph-mon[25218]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:26:53.743 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:53 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:26:53.743 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:53 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:54.642 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:54 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:54.642 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:54 smithi026 ceph-mon[25218]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:26:55.086 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:26:55.086 INFO:teuthology.orchestra.run.smithi026.stdout:[{"addr": "172.21.15.26", "hostname": "smithi026", "labels": [], "status": ""}, {"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}] 2024-03-16T14:26:55.554 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-16T14:26:55.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-16T14:26:55.789 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:26:56.195 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:56 smithi026 ceph-mon[25218]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T14:26:56.196 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:56 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:57 smithi026 ceph-mon[25218]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:26:58.106 INFO:teuthology.orchestra.run.smithi026.stderr:adjusted tunables profile to default 2024-03-16T14:26:58.430 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:58 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1021427520' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T14:26:58.767 INFO:tasks.cephadm:Adding mon.a on smithi026 2024-03-16T14:26:58.767 INFO:tasks.cephadm:Adding mon.c on smithi026 2024-03-16T14:26:58.767 INFO:tasks.cephadm:Adding mon.b on smithi151 2024-03-16T14:26:58.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi026:172.21.15.26=a;smithi026:[v2:172.21.15.26:3301,v1:172.21.15.26:6790]=c;smithi151:172.21.15.151=b' 2024-03-16T14:26:59.000 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:26:59.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1021427520' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T14:26:59.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:26:59.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:59.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:59.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:59.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:26:59.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:26:59.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:26:59.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:26:59.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:26:59.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:26:59.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:26:59 smithi026 ceph-mon[25218]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:00.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:00 smithi026 ceph-mon[25218]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:27:00.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:00 smithi026 ceph-mon[25218]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:27:00.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:00 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:00.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:00 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:00.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:00 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:00.682 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mon update... 2024-03-16T14:27:01.241 DEBUG:teuthology.orchestra.run.smithi026:mon.c> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.c.service 2024-03-16T14:27:01.244 DEBUG:teuthology.orchestra.run.smithi151:mon.b> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.b.service 2024-03-16T14:27:01.246 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T14:27:01.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:02.332 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:02.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:02 smithi026 ceph-mon[25218]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi026:[v2:172.21.15.26:3301,v1:172.21.15.26:6790]=c;smithi151:172.21.15.151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:02.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:02 smithi026 ceph-mon[25218]: Saving service mon spec with placement smithi026:172.21.15.26=a;smithi026:[v2:172.21.15.26:3301,v1:172.21.15.26:6790]=c;smithi151:172.21.15.151=b;count:3 2024-03-16T14:27:02.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:02 smithi026 ceph-mon[25218]: Deploying daemon mon.b on smithi151 2024-03-16T14:27:03.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:03 smithi026 ceph-mon[25218]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:05.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:04 smithi026 ceph-mon[25218]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:05.259 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:27:05.259 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1,"fsid":"efcc9110-e3a0-11ee-95c9-87774f69a715","modified":"2024-03-16T14:25:27.153917Z","created":"2024-03-16T14:25:27.153917Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-16T14:27:05.261 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 1 2024-03-16T14:27:05.363 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:05 smithi151 ceph-mon[30603]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T14:27:05.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:05 smithi151 bash[30571]: 10f194751dd69b2a837123a3eb30f2a998741a414020ea410c5f7fd4ca9de02e 2024-03-16T14:27:05.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:05 smithi151 systemd[1]: Started Ceph mon.b for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:27:06.970 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T14:27:06.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T14:27:07.218 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:27:10.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:27:10.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: mon.a calling monitor election 2024-03-16T14:27:10.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: mon.b calling monitor election 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: monmap e2: 2 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: fsmap 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: mgrmap e13: y(active, since 39s) 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: overall HEALTH_OK 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:10.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:10.740 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:10.740 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:10.740 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: mon.a calling monitor election 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: mon.b calling monitor election 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: monmap e2: 2 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: fsmap 2024-03-16T14:27:10.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:27:10.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: mgrmap e13: y(active, since 39s) 2024-03-16T14:27:10.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: overall HEALTH_OK 2024-03-16T14:27:10.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:10.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:10.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:10.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:10.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:11.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:11 smithi151 ceph-mon[30603]: Deploying daemon mon.c on smithi026 2024-03-16T14:27:11.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:11 smithi151 ceph-mon[30603]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:11.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:11 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:11.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:11 smithi026 ceph-mon[25218]: Deploying daemon mon.c on smithi026 2024-03-16T14:27:11.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:11 smithi026 ceph-mon[25218]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:11.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:11 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:11.901 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:27:11.902 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":2,"fsid":"efcc9110-e3a0-11ee-95c9-87774f69a715","modified":"2024-03-16T14:27:05.355495Z","created":"2024-03-16T14:25:27.153917Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-16T14:27:11.905 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 2 2024-03-16T14:27:12.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:12 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/4115538876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T14:27:12.506 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:27:12 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:27:12.355+0000 7fbc2d4e6700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-16T14:27:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:12 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/4115538876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T14:27:13.461 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-16T14:27:13.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-16T14:27:13.706 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:27:13.721 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:13 smithi026 ceph-mon[25218]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:13.721 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:13 smithi026 podman[38261]: 2024-03-16 14:27:13.718511656 +0000 UTC m=+0.158623447 container create 12f7db96a36003427ad7bb99f2dc6551ac268853a79e022f63007196132ecfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-16T14:27:13.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:13 smithi151 ceph-mon[30603]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.381 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 podman[38261]: 2024-03-16 14:27:14.043412947 +0000 UTC m=+0.483524746 container init 12f7db96a36003427ad7bb99f2dc6551ac268853a79e022f63007196132ecfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c) 2024-03-16T14:27:14.381 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 podman[38261]: 2024-03-16 14:27:14.046970489 +0000 UTC m=+0.487082280 container start 12f7db96a36003427ad7bb99f2dc6551ac268853a79e022f63007196132ecfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c) 2024-03-16T14:27:14.381 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: set uid:gid to 167:167 (ceph:ceph) 2024-03-16T14:27:14.381 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: ceph version 19.0.0-2121-g5ea4d9cf (5ea4d9cfac4d9f2893126a48d28cca7da69243c1) squid (dev), process ceph-mon, pid 2 2024-03-16T14:27:14.381 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pidfile_write: ignore empty --pid-file 2024-03-16T14:27:14.381 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: load: jerasure load: lrc 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: RocksDB version: 7.9.2 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Git sha 0 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Compile date 2024-03-15 15:33:47 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: DB SUMMARY 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: DB Session ID: AQ0J4R14N5G4GJNXMX7M 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: CURRENT file: CURRENT 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: IDENTITY file: IDENTITY 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.error_if_exists: 0 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.create_if_missing: 0 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.paranoid_checks: 1 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-16T14:27:14.382 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-16T14:27:14.383 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-16T14:27:14.383 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.env: 0x55ef85e7bb20 2024-03-16T14:27:14.383 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.fs: PosixFileSystem 2024-03-16T14:27:14.383 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.info_log: 0x55ef88439920 2024-03-16T14:27:14.383 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_file_opening_threads: 16 2024-03-16T14:27:14.384 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.statistics: (nil) 2024-03-16T14:27:14.384 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.use_fsync: 0 2024-03-16T14:27:14.384 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_log_file_size: 0 2024-03-16T14:27:14.384 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.keep_log_file_num: 1000 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.recycle_log_file_num: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.allow_fallocate: 1 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.allow_mmap_reads: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.allow_mmap_writes: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.use_direct_reads: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.create_missing_column_families: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.db_log_dir: 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.wal_dir: 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-16T14:27:14.385 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.advise_random_on_open: 1 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.db_write_buffer_size: 0 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.write_buffer_manager: 0x55ef8840c1e0 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.rate_limiter: (nil) 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.wal_recovery_mode: 2 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.enable_thread_tracking: 0 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.enable_pipelined_write: 0 2024-03-16T14:27:14.386 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.unordered_write: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.row_cache: None 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.wal_filter: None 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.allow_ingest_behind: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.two_write_queues: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.manual_wal_flush: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.wal_compression: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.atomic_flush: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.log_readahead_size: 0 2024-03-16T14:27:14.387 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.best_efforts_recovery: 0 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.allow_data_in_errors: 0 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.db_host_id: __hostname__ 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_background_jobs: 2 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_background_compactions: -1 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_subcompactions: 1 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_total_wal_size: 0 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-16T14:27:14.388 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_open_files: -1 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bytes_per_sync: 0 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_readahead_size: 0 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_background_flushes: -1 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Compression algorithms supported: 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kZSTD supported: 0 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kXpressCompression supported: 0 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kLZ4HCCompression supported: 1 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kZlibCompression supported: 1 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kSnappyCompression supported: 1 2024-03-16T14:27:14.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kLZ4Compression supported: 1 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: kBZip2Compression supported: 0 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.merge_operator: 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_filter: None 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_filter_factory: None 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.sst_partitioner_factory: None 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ef88439a20) 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout: cache_index_and_filter_blocks: 1 2024-03-16T14:27:14.390 INFO:journalctl@ceph.mon.c.smithi026.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: pin_top_level_index_and_filter: 1 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: index_type: 0 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: data_block_index_type: 0 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: index_shortening: 1 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: checksum: 4 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: no_block_cache: 0 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_cache: 0x55ef8843f090 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_cache_name: BinnedLRUCache 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_cache_options: 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: capacity : 536870912 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: num_shard_bits : 4 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: strict_capacity_limit : 0 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: high_pri_pool_ratio: 0.000 2024-03-16T14:27:14.391 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_cache_compressed: (nil) 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: persistent_cache: (nil) 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_size: 4096 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_size_deviation: 10 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_restart_interval: 16 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: index_block_restart_interval: 1 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: metadata_block_size: 4096 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: partition_filters: 0 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: use_delta_encoding: 1 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: filter_policy: bloomfilter 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: whole_key_filtering: 1 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: verify_compression: 0 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: read_amp_bytes_per_bit: 0 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: format_version: 5 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: enable_index_compression: 1 2024-03-16T14:27:14.392 INFO:journalctl@ceph.mon.c.smithi026.stdout: block_align: 0 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout: max_auto_readahead_size: 262144 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout: prepopulate_block_cache: 0 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout: initial_auto_readahead_size: 8192 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout: num_file_reads_for_auto_readahead: 2 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.write_buffer_size: 33554432 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_write_buffer_number: 2 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression: NoCompression 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression: Disabled 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.prefix_extractor: nullptr 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.num_levels: 7 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-16T14:27:14.393 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.level: 32767 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.strategy: 0 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.enabled: false 2024-03-16T14:27:14.394 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.target_file_size_base: 67108864 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-16T14:27:14.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.arena_block_size: 1048576 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.disable_auto_compactions: 0 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-16T14:27:14.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.table_properties_collectors: 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.inplace_update_support: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.bloom_locality: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.max_successive_merges: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.paranoid_file_checks: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.force_consistency_checks: 1 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.report_bg_io_stats: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.ttl: 2592000 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-16T14:27:14.397 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.enable_blob_files: false 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.min_blob_size: 0 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.blob_file_size: 268435456 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.blob_file_starting_level: 0 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 81314bda-cffe-4d12-872f-8916e73e5c91 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710599234076659, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-16T14:27:14.398 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710599234077219, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710599234, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "81314bda-cffe-4d12-872f-8916e73e5c91", "db_session_id": "AQ0J4R14N5G4GJNXMX7M", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710599234077328, "job": 1, "event": "recovery_finished"} 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ef88532000 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: DB pointer 0x55ef8851c000 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: ** DB Stats ** 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-16T14:27:14.399 INFO:journalctl@ceph.mon.c.smithi026.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: ** Compaction Stats [default] ** 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: 2024-03-16T14:27:14.400 INFO:journalctl@ceph.mon.c.smithi026.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: Block cache BinnedLRUCache@0x55ef8843f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-16T14:27:14.401 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: using public_addrv [v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0] 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: starting mon.c rank -1 at public addrs [v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0] at bind addrs [v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(???) e0 preinit fsid efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).mds e1 new map 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout: e1 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout: legacy client fscid: -1 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout: 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout: No filesystems configured 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-16T14:27:14.402 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1021427520' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1021427520' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:27:14.403 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:14.404 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi026:172.21.15.26=a;smithi026:[v2:172.21.15.26:3301,v1:172.21.15.26:6790]=c;smithi151:172.21.15.151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: Saving service mon spec with placement smithi026:172.21.15.26=a;smithi026:[v2:172.21.15.26:3301,v1:172.21.15.26:6790]=c;smithi151:172.21.15.151=b;count:3 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: Deploying daemon mon.b on smithi151 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.a calling monitor election 2024-03-16T14:27:14.405 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.b calling monitor election 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: monmap e2: 2 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: fsmap 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mgrmap e13: y(active, since 39s) 2024-03-16T14:27:14.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: overall HEALTH_OK 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: Deploying daemon mon.c on smithi026 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/4115538876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 ceph-mon[38287]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 bash[38261]: 12f7db96a36003427ad7bb99f2dc6551ac268853a79e022f63007196132ecfea 2024-03-16T14:27:14.407 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:14 smithi026 systemd[1]: Started Ceph mon.c for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: mon.a calling monitor election 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: mon.b calling monitor election 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: mon.c calling monitor election 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],c=[v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: fsmap 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: mgrmap e13: y(active, since 48s) 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: overall HEALTH_OK 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:19.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: mon.a calling monitor election 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: mon.b calling monitor election 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: mon.c calling monitor election 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],c=[v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: fsmap 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: mgrmap e13: y(active, since 48s) 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: overall HEALTH_OK 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:19.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: mon.a calling monitor election 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: mon.b calling monitor election 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: mon.c calling monitor election 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: monmap e3: 3 mons at {a=[v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0],b=[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0],c=[v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: fsmap 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: osdmap e4: 0 total, 0 up, 0 in 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: mgrmap e13: y(active, since 48s) 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: overall HEALTH_OK 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:19.509 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:19.885 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-03-16T14:27:19.885 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":3,"fsid":"efcc9110-e3a0-11ee-95c9-87774f69a715","modified":"2024-03-16T14:27:14.098836Z","created":"2024-03-16T14:25:27.153917Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3300","nonce":0},{"type":"v1","addr":"172.21.15.26:6789","nonce":0}]},"addr":"172.21.15.26:6789/0","public_addr":"172.21.15.26:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:3301","nonce":0},{"type":"v1","addr":"172.21.15.26:6790","nonce":0}]},"addr":"172.21.15.26:6790/0","public_addr":"172.21.15.26:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-16T14:27:19.888 INFO:teuthology.orchestra.run.smithi151.stderr:dumped monmap epoch 3 2024-03-16T14:27:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:20 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:20 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:20 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:20 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/3860620928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T14:27:20.465 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:20 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:20.467 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-16T14:27:20.467 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-16T14:27:20.492 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:20.492 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:20.492 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:20.492 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/3860620928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T14:27:20.492 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:20.493 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:20.493 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:20.493 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:20.493 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/3860620928' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-16T14:27:20.493 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:20 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:27:20.721 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:21.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:21.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:21.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:21.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:21.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:21.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:22.492 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-16T14:27:22.492 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:22.493 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.494 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.495 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:22.495 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:22.495 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:22.596 INFO:teuthology.orchestra.run.smithi026.stdout:# minimal ceph.conf for efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:27:22.597 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-03-16T14:27:22.597 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:27:22.597 INFO:teuthology.orchestra.run.smithi026.stdout: mon_host = [v2:172.21.15.26:3300/0,v1:172.21.15.26:6789/0] [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] [v2:172.21.15.26:3301/0,v1:172.21.15.26:6790/0] 2024-03-16T14:27:22.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:22.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:22.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:22.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:23.123 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-16T14:27:23.123 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:27:23.123 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T14:27:23.192 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:27:23.192 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:27:23.259 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:27:23.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-03-16T14:27:23.294 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:27:23.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:27:23.366 INFO:tasks.cephadm:Adding mgr.y on smithi026 2024-03-16T14:27:23.366 INFO:tasks.cephadm:Adding mgr.x on smithi151 2024-03-16T14:27:23.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply mgr '2;smithi026=y;smithi151=x' 2024-03-16T14:27:23.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[38287]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T14:27:23.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[38287]: Reconfiguring daemon mon.a on smithi026 2024-03-16T14:27:23.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[38287]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:23.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3686632133' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:23.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[25218]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T14:27:23.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[25218]: Reconfiguring daemon mon.a on smithi026 2024-03-16T14:27:23.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[25218]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:23.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:23 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3686632133' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:23.640 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:27:23.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:23 smithi151 ceph-mon[30603]: Reconfiguring mon.a (unknown last config time)... 2024-03-16T14:27:23.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:23 smithi151 ceph-mon[30603]: Reconfiguring daemon mon.a on smithi026 2024-03-16T14:27:23.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:23 smithi151 ceph-mon[30603]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:23.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:23 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3686632133' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:25.161 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled mgr update... 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: Reconfiguring mon.c (monmap changed)... 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: Reconfiguring daemon mon.c on smithi026 2024-03-16T14:27:25.162 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:24 smithi151 ceph-mon[30603]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: Reconfiguring mon.c (monmap changed)... 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: Reconfiguring daemon mon.c on smithi026 2024-03-16T14:27:25.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[25218]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: Reconfiguring mon.c (monmap changed)... 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: Reconfiguring daemon mon.c on smithi026 2024-03-16T14:27:25.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:24 smithi026 ceph-mon[38287]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:25.746 DEBUG:teuthology.orchestra.run.smithi151:mgr.x> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.x.service 2024-03-16T14:27:25.748 INFO:tasks.cephadm:Deploying OSDs... 2024-03-16T14:27:25.748 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:27:25.749 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T14:27:25.765 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-16T14:27:25.765 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/nvme1n1 2024-03-16T14:27:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/nvme1n1 2024-03-16T14:27:25.821 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:25.822 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-16T14:27:25.822 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:25.822 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:25.822 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:26:42.563037343 +0000 2024-03-16T14:27:25.822 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:42.867862642 +0000 2024-03-16T14:27:25.822 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:42.867862642 +0000 2024-03-16T14:27:25.822 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:42.857862790 +0000 2024-03-16T14:27:25.822 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-16T14:27:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:27:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:27:25.891 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000111918 s, 4.6 MB/s 2024-03-16T14:27:25.891 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-16T14:27:25.947 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/nvme2n1 2024-03-16T14:27:26.001 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/nvme2n1 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:26:42.571037224 +0000 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:43.018860399 +0000 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:43.018860399 +0000 2024-03-16T14:27:26.002 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:43.013860473 +0000 2024-03-16T14:27:26.002 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-16T14:27:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:27:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:27:26.068 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000124027 s, 4.1 MB/s 2024-03-16T14:27:26.069 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-16T14:27:26.125 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/nvme3n1 2024-03-16T14:27:26.180 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/nvme3n1 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:26:42.580037090 +0000 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:43.166858201 +0000 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:43.166858201 +0000 2024-03-16T14:27:26.181 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:43.157858335 +0000 2024-03-16T14:27:26.182 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-16T14:27:26.246 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:27:26.247 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:27:26.247 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000118062 s, 4.3 MB/s 2024-03-16T14:27:26.248 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-16T14:27:26.304 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/nvme4n1 2024-03-16T14:27:26.359 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/nvme4n1 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-16 14:26:42.588036972 +0000 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-16 14:24:43.309856077 +0000 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-16 14:24:43.309856077 +0000 2024-03-16T14:27:26.360 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-16 14:24:43.300856211 +0000 2024-03-16T14:27:26.360 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-16T14:27:26.372 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi026=y;smithi151=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: Saving service mgr spec with placement smithi026=y;smithi151=x;count:2 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: Reconfiguring mon.b (monmap changed)... 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:26.373 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:26 smithi151 ceph-mon[30603]: Reconfiguring daemon mon.b on smithi151 2024-03-16T14:27:26.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi026=y;smithi151=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:26.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: Saving service mgr spec with placement smithi026=y;smithi151=x;count:2 2024-03-16T14:27:26.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.423 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: Reconfiguring mon.b (monmap changed)... 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[25218]: Reconfiguring daemon mon.b on smithi151 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi026=y;smithi151=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: Saving service mgr spec with placement smithi026=y;smithi151=x;count:2 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: Reconfiguring mon.b (monmap changed)... 2024-03-16T14:27:26.424 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-16T14:27:26.425 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-16T14:27:26.425 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:26.425 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:26 smithi026 ceph-mon[38287]: Reconfiguring daemon mon.b on smithi151 2024-03-16T14:27:26.425 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-16T14:27:26.425 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-16T14:27:26.425 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000119082 s, 4.3 MB/s 2024-03-16T14:27:26.427 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-16T14:27:26.482 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:27:26.482 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-03-16T14:27:26.499 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-16T14:27:26.499 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme1n1 2024-03-16T14:27:26.553 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme1n1 2024-03-16T14:27:26.553 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:26.553 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-16T14:27:26.553 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:26.554 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:26.554 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:26:57.253297327 +0000 2024-03-16T14:27:26.554 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:44.496905984 +0000 2024-03-16T14:27:26.554 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:44.496905984 +0000 2024-03-16T14:27:26.554 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:44.485906114 +0000 2024-03-16T14:27:26.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-16T14:27:26.619 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:27:26.620 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:27:26.620 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000231697 s, 2.2 MB/s 2024-03-16T14:27:26.621 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-16T14:27:26.677 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme2n1 2024-03-16T14:27:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme2n1 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:26:57.264297197 +0000 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:44.659904052 +0000 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:44.659904052 +0000 2024-03-16T14:27:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:44.651904147 +0000 2024-03-16T14:27:26.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-16T14:27:26.796 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:27:26.796 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:27:26.796 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.00021328 s, 2.4 MB/s 2024-03-16T14:27:26.797 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-16T14:27:26.853 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme3n1 2024-03-16T14:27:26.908 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme3n1 2024-03-16T14:27:26.908 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:26.908 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-16T14:27:26.908 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:26.908 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:26.909 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:26:57.275297066 +0000 2024-03-16T14:27:26.909 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:44.810902263 +0000 2024-03-16T14:27:26.909 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:44.810902263 +0000 2024-03-16T14:27:26.909 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:44.798902405 +0000 2024-03-16T14:27:26.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-16T14:27:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:27:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:27:26.974 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000214568 s, 2.4 MB/s 2024-03-16T14:27:26.976 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-16T14:27:27.032 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/nvme4n1 2024-03-16T14:27:27.087 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/nvme4n1 2024-03-16T14:27:27.087 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-16T14:27:27.087 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-16T14:27:27.087 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-16T14:27:27.088 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-16T14:27:27.088 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-03-16 14:26:57.285296948 +0000 2024-03-16T14:27:27.088 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-03-16 14:24:44.968900390 +0000 2024-03-16T14:27:27.088 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-03-16 14:24:44.968900390 +0000 2024-03-16T14:27:27.088 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-03-16 14:24:44.959900497 +0000 2024-03-16T14:27:27.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-16T14:27:27.154 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-03-16T14:27:27.154 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-03-16T14:27:27.155 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000222698 s, 2.3 MB/s 2024-03-16T14:27:27.156 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-16T14:27:27.211 INFO:tasks.cephadm:Deploying osd.0 on smithi026 with /dev/nvme4n1... 2024-03-16T14:27:27.211 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-16T14:27:27.458 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:27:27.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T14:27:27.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:27 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:27.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T14:27:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-16T14:27:27.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T14:27:27.508 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:27 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:28.467 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:28 smithi151 ceph-mon[30603]: Deploying daemon mgr.x on smithi151 2024-03-16T14:27:28.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:28 smithi026 ceph-mon[25218]: Deploying daemon mgr.x on smithi151 2024-03-16T14:27:28.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:28 smithi026 ceph-mon[38287]: Deploying daemon mgr.x on smithi151 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/nvme4n1 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00926112 s, 1.1 GB/s 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 2024-03-16T14:27:29.342 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-16T14:27:29.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:29 smithi151 ceph-mon[30603]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:29.488 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:29 smithi151 systemd[1]: Started Ceph mgr.x for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:27:29.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:29 smithi026 ceph-mon[25218]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:29.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:29 smithi026 ceph-mon[38287]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:29.970 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:/dev/nvme4n1 2024-03-16T14:27:30.210 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:27:30.237 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:29 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:29.828+0000 7f440abd2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T14:27:30.237 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:29 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:29.926+0000 7f440abd2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T14:27:30.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.591 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:30.591 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.592 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.592 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.592 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.592 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:30.608 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.609 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.609 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.609 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:30.609 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:30.954 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:30.604+0000 7f440abd2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T14:27:30.954 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:30.708+0000 7f440abd2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T14:27:30.954 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:30.854+0000 7f440abd2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T14:27:30.955 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:30 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:30.950+0000 7f440abd2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T14:27:31.237 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:31.033+0000 7f440abd2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T14:27:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T14:27:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:31.587 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:31.297+0000 7f440abd2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:31.404+0000 7f440abd2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:31.490+0000 7f440abd2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:31.583+0000 7f440abd2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:31.588 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:31.589 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-16T14:27:31.589 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-16T14:27:31.589 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:31.987 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:31 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:31.759+0000 7f440abd2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T14:27:32.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:32 smithi026 ceph-mon[25218]: Reconfiguring mgr.y (unknown last config time)... 2024-03-16T14:27:32.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:32 smithi026 ceph-mon[25218]: Reconfiguring daemon mgr.y on smithi026 2024-03-16T14:27:32.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:32 smithi026 ceph-mon[38287]: Reconfiguring mgr.y (unknown last config time)... 2024-03-16T14:27:32.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:32 smithi026 ceph-mon[38287]: Reconfiguring daemon mgr.y on smithi026 2024-03-16T14:27:32.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:32 smithi151 ceph-mon[30603]: Reconfiguring mgr.y (unknown last config time)... 2024-03-16T14:27:32.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:32 smithi151 ceph-mon[30603]: Reconfiguring daemon mgr.y on smithi026 2024-03-16T14:27:33.208 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:32 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:32.735+0000 7f440abd2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T14:27:33.208 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:32 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:32.869+0000 7f440abd2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T14:27:33.208 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:33.204+0000 7f440abd2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T14:27:33.487 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:33.293+0000 7f440abd2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:33.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:33.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:27:33.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:27:33.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:33.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:33.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:33.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:33.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:33.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:33.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:33.962 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:33.488+0000 7f440abd2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:27:33.962 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:33.673+0000 7f440abd2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T14:27:34.222 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:33 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:33.958+0000 7f440abd2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:27:34.223 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:34 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:34.219+0000 7f440abd2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T14:27:34.487 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:34 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:34.454+0000 7f440abd2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T14:27:34.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:34 smithi151 ceph-mon[30603]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:35.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:34 smithi026 ceph-mon[25218]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:35.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:34 smithi026 ceph-mon[38287]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:35.617 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:35.259+0000 7f440abd2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T14:27:35.932 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4288170715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]: dispatch 2024-03-16T14:27:35.932 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]: dispatch 2024-03-16T14:27:35.932 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]': finished 2024-03-16T14:27:35.932 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-mon[30603]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T14:27:35.933 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:35.933 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:35.832+0000 7f440abd2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T14:27:35.933 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:35 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:35.928+0000 7f440abd2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T14:27:36.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4288170715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]: dispatch 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]: dispatch 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]': finished 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[25218]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4288170715' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]: dispatch 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]: dispatch 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e773bb90-a94a-471e-8d8b-51a7cbada97b"}]': finished 2024-03-16T14:27:36.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[38287]: osdmap e5: 1 total, 0 up, 1 in 2024-03-16T14:27:36.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:35 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:36.237 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:27:36.143+0000 7f440abd2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T14:27:36.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-mon[30603]: Standby manager daemon x started 2024-03-16T14:27:36.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-16T14:27:36.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T14:27:36.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-16T14:27:36.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T14:27:36.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3282588729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:27:36.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:36 smithi151 ceph-mon[30603]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[25218]: Standby manager daemon x started 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3282588729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[25218]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[38287]: Standby manager daemon x started 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T14:27:37.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-16T14:27:37.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/4183693478' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T14:27:37.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3282588729' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:27:37.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:36 smithi026 ceph-mon[38287]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:37.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:37 smithi151 ceph-mon[30603]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-16T14:27:37.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T14:27:38.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:37 smithi026 ceph-mon[25218]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-16T14:27:38.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T14:27:38.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:37 smithi026 ceph-mon[38287]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-16T14:27:38.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T14:27:38.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:38 smithi026 ceph-mon[38287]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:38.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:38 smithi026 ceph-mon[25218]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:38.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:38 smithi151 ceph-mon[30603]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:40.919 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:40 smithi026 ceph-mon[25218]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:40.919 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:40 smithi026 ceph-mon[38287]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:40.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:40 smithi151 ceph-mon[30603]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:42.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T14:27:42.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:42.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T14:27:42.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:42.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-16T14:27:42.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:43.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:42 smithi026 ceph-mon[25218]: Deploying daemon osd.0 on smithi026 2024-03-16T14:27:43.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:42 smithi026 ceph-mon[25218]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:43.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:42 smithi026 ceph-mon[38287]: Deploying daemon osd.0 on smithi026 2024-03-16T14:27:43.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:42 smithi026 ceph-mon[38287]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:43.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:43 smithi151 ceph-mon[30603]: Deploying daemon osd.0 on smithi026 2024-03-16T14:27:43.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:43 smithi151 ceph-mon[30603]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:44.735 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:44 smithi026 ceph-mon[25218]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:44.735 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:44 smithi026 ceph-mon[38287]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:44.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:44 smithi151 ceph-mon[30603]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:46.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:46 smithi151 ceph-mon[30603]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:47.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:46 smithi026 ceph-mon[25218]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:47.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:46 smithi026 ceph-mon[38287]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:48.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:48 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:48.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:48 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:48.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:48 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:48.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:48 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:48.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:48 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:48.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:48 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:48.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:48 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:48.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:48 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:48.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:48 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:49.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:49 smithi151 ceph-mon[30603]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:49.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:49 smithi026 ceph-mon[38287]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:49.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:49 smithi026 ceph-mon[25218]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:51.054 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 0 on host 'smithi026' 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.332 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.333 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:51.433 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:51.434 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:27:51.751 DEBUG:teuthology.orchestra.run.smithi026:osd.0> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.0.service 2024-03-16T14:27:51.753 INFO:tasks.cephadm:Deploying osd.1 on smithi026 with /dev/nvme3n1... 2024-03-16T14:27:51.754 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-16T14:27:52.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:52 smithi151 ceph-mon[30603]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T14:27:52.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:52 smithi151 ceph-mon[30603]: from='osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:27:52.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:52 smithi151 ceph-mon[30603]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T14:27:52.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:52 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:52.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:52 smithi151 ceph-mon[30603]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:27:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[25218]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T14:27:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[25218]: from='osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:27:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[25218]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T14:27:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[25218]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:27:52.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[38287]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-16T14:27:52.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[38287]: from='osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:27:52.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[38287]: osdmap e6: 1 total, 0 up, 1 in 2024-03-16T14:27:52.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:52.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-mon[38287]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:27:52.507 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0[42155]: 2024-03-16T14:27:52.155+0000 7ff33c2f3700 -1 osd.0 0 waiting for initial osdmap 2024-03-16T14:27:52.507 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:27:52 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0[42155]: 2024-03-16T14:27:52.157+0000 7ff3380e6700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:27:52.782 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:27:53.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[25218]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:27:53.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[25218]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T14:27:53.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[25218]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:53.434 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.434 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[38287]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:27:53.435 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[38287]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T14:27:53.435 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.435 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.435 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[38287]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:53.435 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:53 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:53 smithi151 ceph-mon[30603]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:27:53.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:53 smithi151 ceph-mon[30603]: osdmap e7: 1 total, 0 up, 1 in 2024-03-16T14:27:53.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:53 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:53 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:53.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:53 smithi151 ceph-mon[30603]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-16T14:27:53.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:53 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:54.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:27:54.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:27:54.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[25218]: osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011] boot 2024-03-16T14:27:54.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[25218]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T14:27:54.442 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:54.442 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:27:54.442 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:27:54.443 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[38287]: osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011] boot 2024-03-16T14:27:54.443 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[38287]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T14:27:54.443 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:54 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:54 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:27:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:54 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:27:54.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:54 smithi151 ceph-mon[30603]: osd.0 [v2:172.21.15.26:6802/2042975011,v1:172.21.15.26:6803/2042975011] boot 2024-03-16T14:27:54.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:54 smithi151 ceph-mon[30603]: osdmap e8: 1 total, 1 up, 1 in 2024-03-16T14:27:54.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:54 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:27:54.806 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/nvme3n1 2024-03-16T14:27:54.806 INFO:teuthology.orchestra.run.smithi026.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:27:54.806 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-16T14:27:54.806 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-16T14:27:54.806 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-16T14:27:54.806 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00957989 s, 1.1 GB/s 2024-03-16T14:27:54.807 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-16T14:27:55.487 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:55 smithi026 ceph-mon[38287]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T14:27:55.487 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:55 smithi026 ceph-mon[38287]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:55.487 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:55 smithi026 ceph-mon[25218]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T14:27:55.488 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:55 smithi026 ceph-mon[25218]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:55.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:55 smithi151 ceph-mon[30603]: osdmap e9: 1 total, 1 up, 1 in 2024-03-16T14:27:55.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:55 smithi151 ceph-mon[30603]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:55.513 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:/dev/nvme3n1 2024-03-16T14:27:56.037 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi026 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi026 to 16069M 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[38287]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi026 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi026 to 16069M 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.257 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:57 smithi026 ceph-mon[25218]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:57.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi026 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi026 to 16069M 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:27:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:57 smithi151 ceph-mon[30603]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:58.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:27:58.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:27:58.409 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:58.409 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:27:58.409 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:27:58.409 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:27:58.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:27:58.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:27:59.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:59 smithi026 ceph-mon[25218]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:59.343 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:27:59 smithi026 ceph-mon[25218]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:59.343 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:59 smithi026 ceph-mon[38287]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:59.343 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:27:59 smithi026 ceph-mon[38287]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:27:59.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:59 smithi151 ceph-mon[30603]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:27:59.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:27:59 smithi151 ceph-mon[30603]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:00.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:00 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1596807538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79b74b66-d263-4eb1-af4c-76fcef030e09"}]: dispatch 2024-03-16T14:28:00.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:00 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1596807538' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79b74b66-d263-4eb1-af4c-76fcef030e09"}]': finished 2024-03-16T14:28:00.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:00 smithi151 ceph-mon[30603]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T14:28:00.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:00 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1596807538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79b74b66-d263-4eb1-af4c-76fcef030e09"}]: dispatch 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1596807538' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79b74b66-d263-4eb1-af4c-76fcef030e09"}]': finished 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[25218]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1596807538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79b74b66-d263-4eb1-af4c-76fcef030e09"}]: dispatch 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1596807538' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79b74b66-d263-4eb1-af4c-76fcef030e09"}]': finished 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[38287]: osdmap e10: 2 total, 1 up, 2 in 2024-03-16T14:28:00.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:00 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:01.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:01 smithi151 ceph-mon[30603]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:01.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:01 smithi026 ceph-mon[25218]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:01.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:01 smithi026 ceph-mon[38287]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:02.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:02 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/853939945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:02.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:02 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/853939945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:02.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:02 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/853939945' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:03.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:03 smithi151 ceph-mon[30603]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:03.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:03 smithi026 ceph-mon[25218]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:03.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:03 smithi026 ceph-mon[38287]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:04.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:04 smithi026 ceph-mon[38287]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:04.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:04 smithi026 ceph-mon[25218]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:04 smithi151 ceph-mon[30603]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:06.682 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:06 smithi026 ceph-mon[25218]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:06.682 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:06 smithi026 ceph-mon[38287]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:06.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:06 smithi151 ceph-mon[30603]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:07.884 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T14:28:07.884 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:07.884 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:07 smithi026 ceph-mon[38287]: Deploying daemon osd.1 on smithi026 2024-03-16T14:28:07.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T14:28:07.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:07.884 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:07 smithi026 ceph-mon[25218]: Deploying daemon osd.1 on smithi026 2024-03-16T14:28:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-16T14:28:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:07 smithi151 ceph-mon[30603]: Deploying daemon osd.1 on smithi026 2024-03-16T14:28:08.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:08 smithi026 ceph-mon[25218]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:08.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:08 smithi026 ceph-mon[38287]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:08.990 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:08 smithi151 ceph-mon[30603]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:10.912 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:10 smithi026 ceph-mon[38287]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:10.913 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:10 smithi026 ceph-mon[25218]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:10.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:10 smithi151 ceph-mon[30603]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:12.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:12 smithi151 ceph-mon[30603]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:12.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:12.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:12.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:12 smithi026 ceph-mon[25218]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:12.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:12.993 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:12.994 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:12 smithi026 ceph-mon[38287]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:12.994 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:12.994 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:13.982 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:13 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:13.983 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:13 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:13.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:13 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:14.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:14 smithi026 ceph-mon[25218]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:14.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:14 smithi026 ceph-mon[38287]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:14.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:14 smithi151 ceph-mon[30603]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:16.105 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 1 on host 'smithi026' 2024-03-16T14:28:16.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:16.395 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T14:28:16.396 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T14:28:16.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.397 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:16 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:16.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:16.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T14:28:16.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-16T14:28:16.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:16 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:16.705 DEBUG:teuthology.orchestra.run.smithi026:osd.1> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.1.service 2024-03-16T14:28:16.708 INFO:tasks.cephadm:Deploying osd.2 on smithi026 with /dev/nvme2n1... 2024-03-16T14:28:16.709 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-16T14:28:17.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:17 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:17.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:17 smithi151 ceph-mon[30603]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T14:28:17.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:17 smithi151 ceph-mon[30603]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T14:28:17.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:17 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:17.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:17 smithi151 ceph-mon[30603]: from='osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:17.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:17 smithi151 ceph-mon[30603]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:17.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:17 smithi151 ceph-mon[30603]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:17.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[25218]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[25218]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[25218]: from='osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[25218]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[25218]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[38287]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[38287]: osdmap e11: 2 total, 1 up, 2 in 2024-03-16T14:28:17.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:17.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[38287]: from='osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:17.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[38287]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:17.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-mon[38287]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:17.507 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1[50054]: 2024-03-16T14:28:17.127+0000 7f4636378700 -1 osd.1 0 waiting for initial osdmap 2024-03-16T14:28:17.507 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:17 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1[50054]: 2024-03-16T14:28:17.130+0000 7f463216b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:28:17.806 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:28:18.229 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[38287]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:28:18.229 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[38287]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T14:28:18.229 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:18.229 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:18.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[25218]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:28:18.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[25218]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T14:28:18.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:18.230 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:18 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:18.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:18 smithi151 ceph-mon[30603]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:28:18.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:18 smithi151 ceph-mon[30603]: osdmap e12: 2 total, 1 up, 2 in 2024-03-16T14:28:18.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:18 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:18.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:18 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:19.451 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:28:19.451 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:28:19.451 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[38287]: osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532] boot 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[38287]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[38287]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[25218]: osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532] boot 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[25218]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:19.452 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:19 smithi026 ceph-mon[25218]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:19.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:19 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:28:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:19 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:28:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:19 smithi151 ceph-mon[30603]: osd.1 [v2:172.21.15.26:6810/2687555532,v1:172.21.15.26:6811/2687555532] boot 2024-03-16T14:28:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:19 smithi151 ceph-mon[30603]: osdmap e13: 2 total, 2 up, 2 in 2024-03-16T14:28:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:28:19.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:19 smithi151 ceph-mon[30603]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-16T14:28:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/nvme2n1 2024-03-16T14:28:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:28:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-16T14:28:20.162 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-16T14:28:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-16T14:28:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100896 s, 1.0 GB/s 2024-03-16T14:28:20.162 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-16T14:28:20.470 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:20 smithi026 ceph-mon[38287]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T14:28:20.470 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:20 smithi026 ceph-mon[25218]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T14:28:20.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:20 smithi151 ceph-mon[30603]: osdmap e14: 2 total, 2 up, 2 in 2024-03-16T14:28:21.199 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.200 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:21.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:21.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:21.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:21.201 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.227 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:/dev/nvme2n1 2024-03-16T14:28:21.469 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:28:21.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:21.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:21.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:21.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:21.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:21.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:21.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:22.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:22 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi026 2024-03-16T14:28:22.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:22 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi026 to 8034M 2024-03-16T14:28:22.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:22 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi026 2024-03-16T14:28:22.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:22 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi026 to 8034M 2024-03-16T14:28:22.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:22 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi026 2024-03-16T14:28:22.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:22 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi026 to 8034M 2024-03-16T14:28:23.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:23 smithi151 ceph-mon[30603]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:23.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:23 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:28:23.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:23 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:28:23.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:23 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:23.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[38287]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:23.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:28:23.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:28:23.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:23.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[25218]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:23.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:28:23.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:28:23.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:23 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:24.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:24 smithi026 ceph-mon[25218]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:28:24.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:24 smithi026 ceph-mon[38287]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:28:24.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:24 smithi151 ceph-mon[30603]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:28:25.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:25 smithi151 ceph-mon[30603]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:25.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:25 smithi026 ceph-mon[25218]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:25.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:25 smithi026 ceph-mon[38287]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:26.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:26 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/967352788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a33de3b-ab23-4c5f-aeef-5688c34717db"}]: dispatch 2024-03-16T14:28:26.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:26 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/967352788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a33de3b-ab23-4c5f-aeef-5688c34717db"}]': finished 2024-03-16T14:28:26.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:26 smithi151 ceph-mon[30603]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T14:28:26.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:26.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/967352788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a33de3b-ab23-4c5f-aeef-5688c34717db"}]: dispatch 2024-03-16T14:28:26.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/967352788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a33de3b-ab23-4c5f-aeef-5688c34717db"}]': finished 2024-03-16T14:28:26.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[25218]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T14:28:26.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:26.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/967352788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a33de3b-ab23-4c5f-aeef-5688c34717db"}]: dispatch 2024-03-16T14:28:26.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/967352788' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a33de3b-ab23-4c5f-aeef-5688c34717db"}]': finished 2024-03-16T14:28:26.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[38287]: osdmap e15: 3 total, 2 up, 3 in 2024-03-16T14:28:26.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:27.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:27 smithi151 ceph-mon[30603]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:27.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:27 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/2665961122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:27.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:27 smithi026 ceph-mon[38287]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:27.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:27 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/2665961122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:27 smithi026 ceph-mon[25218]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:27.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:27 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2665961122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:28 smithi151 ceph-mon[30603]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:29.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:28 smithi026 ceph-mon[25218]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:29.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:28 smithi026 ceph-mon[38287]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:30 smithi151 ceph-mon[30603]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:30 smithi026 ceph-mon[25218]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:31.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:30 smithi026 ceph-mon[38287]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:32.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[38287]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:32.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T14:28:32.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:32.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[38287]: Deploying daemon osd.2 on smithi026 2024-03-16T14:28:32.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[25218]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:32.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T14:28:32.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:32.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:32 smithi026 ceph-mon[25218]: Deploying daemon osd.2 on smithi026 2024-03-16T14:28:32.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:32 smithi151 ceph-mon[30603]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:32.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:32 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-16T14:28:32.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:32 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:32.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:32 smithi151 ceph-mon[30603]: Deploying daemon osd.2 on smithi026 2024-03-16T14:28:34.845 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:34 smithi026 ceph-mon[38287]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:34.845 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:34 smithi026 ceph-mon[25218]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:34.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:34 smithi151 ceph-mon[30603]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:36.953 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:36 smithi026 ceph-mon[38287]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:36.953 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:36 smithi026 ceph-mon[25218]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:36.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:36 smithi151 ceph-mon[30603]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:39.154 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:39.154 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:39.154 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:39.154 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[38287]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:39.155 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:39.155 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:39.155 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:39.155 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:38 smithi026 ceph-mon[25218]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:39.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:38 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:39.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:38 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:39.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:38 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:39.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:38 smithi151 ceph-mon[30603]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:41.129 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 2 on host 'smithi026' 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:41.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.793 DEBUG:teuthology.orchestra.run.smithi026:osd.2> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.2.service 2024-03-16T14:28:41.795 INFO:tasks.cephadm:Deploying osd.3 on smithi026 with /dev/nvme1n1... 2024-03-16T14:28:41.796 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-16T14:28:41.940 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:28:41.941 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:42.581 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[25218]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T14:28:42.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[25218]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T14:28:42.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:42.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[25218]: from='osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:42.582 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[25218]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:42.582 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[38287]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T14:28:42.582 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[38287]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T14:28:42.582 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:42.583 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[38287]: from='osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:42.583 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-mon[38287]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:42.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:42 smithi151 ceph-mon[30603]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-16T14:28:42.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:42 smithi151 ceph-mon[30603]: osdmap e16: 3 total, 2 up, 3 in 2024-03-16T14:28:42.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:42 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:42.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:42 smithi151 ceph-mon[30603]: from='osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:42.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:42 smithi151 ceph-mon[30603]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:28:43.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2[57988]: 2024-03-16T14:28:42.830+0000 7f030c795700 -1 osd.2 0 waiting for initial osdmap 2024-03-16T14:28:43.177 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:42 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2[57988]: 2024-03-16T14:28:42.833+0000 7f0304580700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:28:43.290 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:28:43.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[25218]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[25218]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[25218]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[25218]: from='osd.2 ' entity='osd.2' 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[38287]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[38287]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[38287]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:43.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:43 smithi026 ceph-mon[38287]: from='osd.2 ' entity='osd.2' 2024-03-16T14:28:43.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:43 smithi151 ceph-mon[30603]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:28:43.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:43 smithi151 ceph-mon[30603]: osdmap e17: 3 total, 2 up, 3 in 2024-03-16T14:28:43.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:43 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:43.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:43 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:43.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:43 smithi151 ceph-mon[30603]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-16T14:28:43.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:43 smithi151 ceph-mon[30603]: from='osd.2 ' entity='osd.2' 2024-03-16T14:28:44.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:28:44.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[38287]: osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318] boot 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[38287]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[25218]: osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318] boot 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[25218]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T14:28:44.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:44 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:44.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:44 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:28:44.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:44 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:28:44.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:44 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:44.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:44 smithi151 ceph-mon[30603]: osd.2 [v2:172.21.15.26:6818/4025043318,v1:172.21.15.26:6819/4025043318] boot 2024-03-16T14:28:44.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:44 smithi151 ceph-mon[30603]: osdmap e18: 3 total, 3 up, 3 in 2024-03-16T14:28:44.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:44 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: pgmap v75: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.556 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.557 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:45.557 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:45.963 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping: /dev/nvme1n1 2024-03-16T14:28:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:28:45.963 INFO:teuthology.orchestra.run.smithi026.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-16T14:28:45.964 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10+0 records in 2024-03-16T14:28:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:10+0 records out 2024-03-16T14:28:45.964 INFO:teuthology.orchestra.run.smithi026.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0099927 s, 1.0 GB/s 2024-03-16T14:28:45.964 INFO:teuthology.orchestra.run.smithi026.stderr:--> Zapping successful for: 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: pgmap v75: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.964 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.965 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.965 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.965 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.965 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:45.965 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:45.965 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:45 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: pgmap v75: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:45.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-16T14:28:45.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-16T14:28:45.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: osdmap e19: 3 total, 3 up, 3 in 2024-03-16T14:28:45.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:28:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:45 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:46.528 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi026:/dev/nvme1n1 2024-03-16T14:28:46.777 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:28:46.886 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi026 2024-03-16T14:28:46.886 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi026 to 5356M 2024-03-16T14:28:46.886 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T14:28:46.886 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[38287]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T14:28:46.890 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi026 2024-03-16T14:28:46.890 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi026 to 5356M 2024-03-16T14:28:46.890 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T14:28:46.890 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:46 smithi026 ceph-mon[25218]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T14:28:46.891 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64028]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-16T14:28:46.891 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64028]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:46.891 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64028]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:46.891 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64028]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:46.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:46 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi026 2024-03-16T14:28:46.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:46 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi026 to 5356M 2024-03-16T14:28:46.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:46 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-16T14:28:46.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:46 smithi151 ceph-mon[30603]: osdmap e20: 3 total, 3 up, 3 in 2024-03-16T14:28:47.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64084]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-16T14:28:47.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64084]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:47.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64084]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:47.196 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:28:46 smithi026 sudo[64084]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:47.505 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64103]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme3n1 2024-03-16T14:28:47.505 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:47.505 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64103]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:47.506 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64103]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:47.794 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[25218]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:47.795 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[25218]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T14:28:47.795 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:47.795 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:47.795 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[38287]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:47.795 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[38287]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T14:28:47.795 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:47.795 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:47 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:47.796 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64110]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme3n1 2024-03-16T14:28:47.796 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64110]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:47.796 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64110]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:47.796 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64110]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:47.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:47 smithi151 ceph-mon[30603]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:47.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:47 smithi151 ceph-mon[30603]: osdmap e21: 3 total, 3 up, 3 in 2024-03-16T14:28:47.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:47 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:47.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:47 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:48.053 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2024-03-16T14:28:48.054 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:48.054 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64113]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:48.054 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:47 smithi026 sudo[64113]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:48.320 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64137]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme2n1 2024-03-16T14:28:48.320 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64137]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:48.320 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64137]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:48.320 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64137]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:48.611 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64146]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T14:28:48.611 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64146]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:48.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64146]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:48.612 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64146]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:49.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64170]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T14:28:49.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64170]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:49.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64170]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:49.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:48 smithi026 sudo[64170]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:49.132 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:48 smithi151 sudo[33153]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T14:28:49.132 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:48 smithi151 sudo[33153]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:49.132 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:48 smithi151 sudo[33153]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:49.132 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:48 smithi151 sudo[33153]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:49.414 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-16T14:28:49.414 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:49.414 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:49.414 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:49.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:49.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:49.417 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:49.417 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:49.417 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:49.417 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64300]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-16T14:28:49.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 sudo[33156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 sudo[33156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 sudo[33156]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 sudo[33156]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:49.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:49.489 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:49.679 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64300]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:49.679 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64300]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:49.679 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64300]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:49.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64315]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-16T14:28:49.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64315]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-16T14:28:49.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64315]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-16T14:28:49.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:49 smithi026 sudo[64315]: pam_unix(sudo:session): session closed for user root 2024-03-16T14:28:50.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[25218]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:50.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:50.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:50.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:50.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:50.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:50.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:50.257 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:50 smithi026 ceph-mon[38287]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:50.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi026:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:28:50.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:50 smithi151 ceph-mon[30603]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-16T14:28:51.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:51 smithi151 ceph-mon[30603]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:51.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:51 smithi026 ceph-mon[25218]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:51.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:51 smithi026 ceph-mon[38287]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:52.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:52 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/814969911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b12f1441-6a57-485e-8bbc-7ae868ed9d43"}]: dispatch 2024-03-16T14:28:52.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:52 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/814969911' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b12f1441-6a57-485e-8bbc-7ae868ed9d43"}]': finished 2024-03-16T14:28:52.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:52 smithi151 ceph-mon[30603]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T14:28:52.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:52 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:28:52.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/814969911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b12f1441-6a57-485e-8bbc-7ae868ed9d43"}]: dispatch 2024-03-16T14:28:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/814969911' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b12f1441-6a57-485e-8bbc-7ae868ed9d43"}]': finished 2024-03-16T14:28:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[25218]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T14:28:52.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:28:52.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/814969911' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b12f1441-6a57-485e-8bbc-7ae868ed9d43"}]: dispatch 2024-03-16T14:28:52.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/814969911' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b12f1441-6a57-485e-8bbc-7ae868ed9d43"}]': finished 2024-03-16T14:28:52.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[38287]: osdmap e22: 4 total, 3 up, 4 in 2024-03-16T14:28:52.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:52 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:28:53.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:53 smithi151 ceph-mon[30603]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:53.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:53 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3649270192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:53.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:53 smithi026 ceph-mon[25218]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:53.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:53 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3649270192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:53.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:53 smithi026 ceph-mon[38287]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:28:53.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:53 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3649270192' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:28:54.894 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:54 smithi026 ceph-mon[38287]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-16T14:28:54.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:54 smithi026 ceph-mon[25218]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-16T14:28:54.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:54 smithi151 ceph-mon[30603]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-16T14:28:56.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:56 smithi026 ceph-mon[25218]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-16T14:28:56.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:56 smithi026 ceph-mon[38287]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-16T14:28:56.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:56 smithi151 ceph-mon[30603]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-16T14:28:58.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T14:28:58.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T14:28:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:58.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-16T14:28:58.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:28:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:59 smithi026 ceph-mon[25218]: Deploying daemon osd.3 on smithi026 2024-03-16T14:28:59.722 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:28:59 smithi026 ceph-mon[25218]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-16T14:28:59.722 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:59 smithi026 ceph-mon[38287]: Deploying daemon osd.3 on smithi026 2024-03-16T14:28:59.722 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:28:59 smithi026 ceph-mon[38287]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-16T14:28:59.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:59 smithi151 ceph-mon[30603]: Deploying daemon osd.3 on smithi026 2024-03-16T14:28:59.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:28:59 smithi151 ceph-mon[30603]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-16T14:29:00.653 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:00 smithi026 ceph-mon[38287]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-16T14:29:00.653 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:00 smithi026 ceph-mon[25218]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-16T14:29:00.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:00 smithi151 ceph-mon[30603]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-16T14:29:02.835 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:02 smithi026 ceph-mon[25218]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-16T14:29:02.836 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:02 smithi026 ceph-mon[38287]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-16T14:29:02.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:02 smithi151 ceph-mon[30603]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 70 KiB/s wr, 5 op/s 2024-03-16T14:29:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:04 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:04 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:04 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:04 smithi151 ceph-mon[30603]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-16T14:29:05.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:05.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:05.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:05.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[25218]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-16T14:29:05.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:05.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:05.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:05.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:04 smithi026 ceph-mon[38287]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-16T14:29:07.415 INFO:teuthology.orchestra.run.smithi026.stdout:Created osd(s) 3 on host 'smithi026' 2024-03-16T14:29:07.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:29:07.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: from='osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T14:29:07.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T14:29:07.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:07.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:07.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:29:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: from='osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T14:29:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T14:29:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: from='osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:08.025 DEBUG:teuthology.orchestra.run.smithi026:osd.3> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.3.service 2024-03-16T14:29:08.028 INFO:tasks.cephadm:Deploying osd.4 on smithi151 with /dev/nvme4n1... 2024-03-16T14:29:08.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-16T14:29:08.273 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:29:08.657 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T14:29:08.658 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T14:29:08.658 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: from='osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:29:08.658 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:08.658 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:29:08.658 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:08.658 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:08.658 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:08 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:08.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T14:29:08.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: from='osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: osdmap e23: 4 total, 3 up, 4 in 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: from='osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:08.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]: dispatch 2024-03-16T14:29:08.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:08.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:08.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:08.757 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:29:08.378+0000 7f47b1031700 -1 osd.3 0 waiting for initial osdmap 2024-03-16T14:29:08.757 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:29:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:29:08.382+0000 7f47ac623700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:29:09.414 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[38287]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:29:09.414 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[38287]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T14:29:09.414 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.414 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.414 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[38287]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:29:09.414 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[25218]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:29:09.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[25218]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T14:29:09.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[25218]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:29:09.415 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:09 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.548 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:09 smithi151 ceph-mon[30603]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi026", "root=default"]}]': finished 2024-03-16T14:29:09.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:09 smithi151 ceph-mon[30603]: osdmap e24: 4 total, 3 up, 4 in 2024-03-16T14:29:09.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:09 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:09 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:09.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:09 smithi151 ceph-mon[30603]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-16T14:29:09.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:09 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:10.076 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/nvme4n1 2024-03-16T14:29:10.076 INFO:teuthology.orchestra.run.smithi151.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:29:10.076 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-16T14:29:10.076 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-16T14:29:10.076 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-16T14:29:10.076 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126104 s, 832 MB/s 2024-03-16T14:29:10.076 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-16T14:29:10.670 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:10 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:29:10.670 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:10 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:29:10.670 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:10 smithi151 ceph-mon[30603]: osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319] boot 2024-03-16T14:29:10.670 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:10 smithi151 ceph-mon[30603]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T14:29:10.670 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:10 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:10.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi151:/dev/nvme4n1 2024-03-16T14:29:10.747 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:29:10.747 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:29:10.747 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[25218]: osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319] boot 2024-03-16T14:29:10.747 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[25218]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T14:29:10.747 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:10.747 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:29:10.748 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:29:10.748 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[38287]: osd.3 [v2:172.21.15.26:6826/3222197319,v1:172.21.15.26:6827/3222197319] boot 2024-03-16T14:29:10.748 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[38287]: osdmap e25: 4 total, 4 up, 4 in 2024-03-16T14:29:10.748 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:10 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:29:10.932 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:29:11.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:11 smithi151 ceph-mon[30603]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T14:29:11.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:11 smithi151 ceph-mon[30603]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:11.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:11 smithi026 ceph-mon[25218]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T14:29:11.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:11 smithi026 ceph-mon[25218]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:11.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:11 smithi026 ceph-mon[38287]: osdmap e26: 4 total, 4 up, 4 in 2024-03-16T14:29:11.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:11 smithi026 ceph-mon[38287]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi026 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi026 to 4017M 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:12.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:12.739 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi026 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi026 to 4017M 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.743 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi026 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi026 to 4017M 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:12.744 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:12.745 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:12.745 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:13.451 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:13 smithi151 ceph-mon[30603]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:13.451 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:13 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:29:13.451 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:13 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:29:13.452 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:13 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:13.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[25218]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:13.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:29:13.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:29:13.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:13.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[38287]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:13.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:29:13.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:29:13.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:13 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:14.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:14 smithi151 ceph-mon[30603]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:29:14.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:14 smithi026 ceph-mon[25218]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:29:14.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:14 smithi026 ceph-mon[38287]: from='client.24187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:29:15.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:15 smithi151 ceph-mon[30603]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:15.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:15 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/3673363884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]: dispatch 2024-03-16T14:29:15.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:15 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]: dispatch 2024-03-16T14:29:15.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:15 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]': finished 2024-03-16T14:29:15.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:15 smithi151 ceph-mon[30603]: osdmap e27: 5 total, 4 up, 5 in 2024-03-16T14:29:15.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:15 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:15.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[25218]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:15.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/3673363884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]: dispatch 2024-03-16T14:29:15.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]: dispatch 2024-03-16T14:29:15.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]': finished 2024-03-16T14:29:15.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[25218]: osdmap e27: 5 total, 4 up, 5 in 2024-03-16T14:29:15.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:15.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[38287]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:15.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/3673363884' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]: dispatch 2024-03-16T14:29:15.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]: dispatch 2024-03-16T14:29:15.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9abf790f-4b0c-4f91-893b-356b795f985d"}]': finished 2024-03-16T14:29:15.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[38287]: osdmap e27: 5 total, 4 up, 5 in 2024-03-16T14:29:15.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:15 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:16.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:16 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/2981615008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:29:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:16 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/2981615008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:29:16.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:16 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/2981615008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:29:17.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:17 smithi151 ceph-mon[30603]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:17.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:17 smithi026 ceph-mon[25218]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:17.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:17 smithi026 ceph-mon[38287]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:18.894 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:18 smithi151 ceph-mon[30603]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:19.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:18 smithi026 ceph-mon[25218]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:19.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:18 smithi026 ceph-mon[38287]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:20.925 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:20 smithi151 ceph-mon[30603]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:20 smithi026 ceph-mon[25218]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:21.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:20 smithi026 ceph-mon[38287]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:22.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T14:29:22.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:22.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T14:29:22.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:22.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-16T14:29:22.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:23.199 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:23 smithi151 ceph-mon[30603]: Deploying daemon osd.4 on smithi151 2024-03-16T14:29:23.199 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:23 smithi151 ceph-mon[30603]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:23.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:23 smithi026 ceph-mon[25218]: Deploying daemon osd.4 on smithi151 2024-03-16T14:29:23.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:23 smithi026 ceph-mon[25218]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:23.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:23 smithi026 ceph-mon[38287]: Deploying daemon osd.4 on smithi151 2024-03-16T14:29:23.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:23 smithi026 ceph-mon[38287]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:24.754 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:24 smithi151 ceph-mon[30603]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:24 smithi026 ceph-mon[25218]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:25.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:24 smithi026 ceph-mon[38287]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:26.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:26 smithi151 ceph-mon[30603]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:27.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:26 smithi026 ceph-mon[38287]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:27.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:26 smithi026 ceph-mon[25218]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:28.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:28 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:28.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:28 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:28.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:28 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:28.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:28 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:28.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:28 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:28.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:28 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:28.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:28 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:28.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:28 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:28.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:28 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:29.579 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:29 smithi151 ceph-mon[30603]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:29.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:29 smithi026 ceph-mon[38287]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:29.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:29 smithi026 ceph-mon[25218]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:31.087 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 4 on host 'smithi151' 2024-03-16T14:29:31.388 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.389 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:31.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T14:29:31.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:31.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:31.507 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:31.739 DEBUG:teuthology.orchestra.run.smithi151:osd.4> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.4.service 2024-03-16T14:29:31.742 INFO:tasks.cephadm:Deploying osd.5 on smithi151 with /dev/nvme3n1... 2024-03-16T14:29:31.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-16T14:29:32.368 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:32 smithi151 ceph-mon[30603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:32 smithi151 ceph-mon[30603]: osdmap e28: 5 total, 4 up, 5 in 2024-03-16T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:32 smithi151 ceph-mon[30603]: from='osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:32 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:32.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:32 smithi151 ceph-mon[30603]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:32.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[25218]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T14:29:32.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[25218]: osdmap e28: 5 total, 4 up, 5 in 2024-03-16T14:29:32.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[25218]: from='osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:32.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:32.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[25218]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:32.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[38287]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-16T14:29:32.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[38287]: osdmap e28: 5 total, 4 up, 5 in 2024-03-16T14:29:32.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[38287]: from='osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:32.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:32.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:32 smithi026 ceph-mon[38287]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:32.921 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:29:32 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:29:32.571+0000 7f0f21e2b700 -1 osd.4 0 waiting for initial osdmap 2024-03-16T14:29:32.922 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:29:32 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:29:32.577+0000 7f0f19c16700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:29:33.210 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:33 smithi151 ceph-mon[30603]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:29:33.210 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:33 smithi151 ceph-mon[30603]: osdmap e29: 5 total, 4 up, 5 in 2024-03-16T14:29:33.210 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:33.210 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:33.210 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:33 smithi151 ceph-mon[30603]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:33.210 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:33 smithi151 ceph-mon[30603]: from='osd.4 ' entity='osd.4' 2024-03-16T14:29:33.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[25218]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[25218]: osdmap e29: 5 total, 4 up, 5 in 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[25218]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[25218]: from='osd.4 ' entity='osd.4' 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[38287]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[38287]: osdmap e29: 5 total, 4 up, 5 in 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:33.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[38287]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-16T14:29:33.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:33 smithi026 ceph-mon[38287]: from='osd.4 ' entity='osd.4' 2024-03-16T14:29:34.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:34 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:29:34.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:34 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:29:34.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:34 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:34.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:34 smithi151 ceph-mon[30603]: osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760] boot 2024-03-16T14:29:34.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:34 smithi151 ceph-mon[30603]: osdmap e30: 5 total, 5 up, 5 in 2024-03-16T14:29:34.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:34 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:34.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:29:34.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:29:34.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[25218]: osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760] boot 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[25218]: osdmap e30: 5 total, 5 up, 5 in 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[38287]: osd.4 [v2:172.21.15.151:6800/1799636760,v1:172.21.15.151:6801/1799636760] boot 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[38287]: osdmap e30: 5 total, 5 up, 5 in 2024-03-16T14:29:34.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:34 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:29:34.992 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/nvme3n1 2024-03-16T14:29:34.992 INFO:teuthology.orchestra.run.smithi151.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:29:34.992 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-16T14:29:34.992 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-16T14:29:34.992 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-16T14:29:34.992 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128616 s, 815 MB/s 2024-03-16T14:29:34.993 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-16T14:29:35.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:35 smithi151 ceph-mon[30603]: osdmap e31: 5 total, 5 up, 5 in 2024-03-16T14:29:35.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:35 smithi151 ceph-mon[30603]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:35.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:35 smithi026 ceph-mon[25218]: osdmap e31: 5 total, 5 up, 5 in 2024-03-16T14:29:35.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:35 smithi026 ceph-mon[25218]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:35.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:35 smithi026 ceph-mon[38287]: osdmap e31: 5 total, 5 up, 5 in 2024-03-16T14:29:35.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:35 smithi026 ceph-mon[38287]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:35.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi151:/dev/nvme3n1 2024-03-16T14:29:36.458 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:29:36.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:36 smithi151 ceph-mon[30603]: osdmap e32: 5 total, 5 up, 5 in 2024-03-16T14:29:36.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:36 smithi026 ceph-mon[25218]: osdmap e32: 5 total, 5 up, 5 in 2024-03-16T14:29:36.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:36 smithi026 ceph-mon[38287]: osdmap e32: 5 total, 5 up, 5 in 2024-03-16T14:29:37.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi151 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi151 to 17088M 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.238 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:37 smithi151 ceph-mon[30603]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T14:29:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi151 2024-03-16T14:29:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi151 to 17088M 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[25218]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: osdmap e33: 5 total, 5 up, 5 in 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi151 2024-03-16T14:29:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi151 to 17088M 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:37.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:37 smithi026 ceph-mon[38287]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:38.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:38 smithi151 ceph-mon[30603]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T14:29:38.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:38 smithi026 ceph-mon[25218]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T14:29:38.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:38 smithi026 ceph-mon[38287]: osdmap e34: 5 total, 5 up, 5 in 2024-03-16T14:29:39.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:39 smithi151 ceph-mon[30603]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:29:39.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:39 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:29:39.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:39 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:29:39.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:39 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:39.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:39 smithi151 ceph-mon[30603]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:39.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[25218]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:29:39.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:29:39.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:29:39.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:39.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[25218]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:39.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[38287]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:29:39.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:29:39.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:29:39.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:39.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:39 smithi026 ceph-mon[38287]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:40.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:40 smithi151 ceph-mon[30603]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:29:41.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:40 smithi026 ceph-mon[25218]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:29:41.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:40 smithi026 ceph-mon[38287]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:29:41.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:41 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/4112262032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]: dispatch 2024-03-16T14:29:41.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:41 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]: dispatch 2024-03-16T14:29:41.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:41 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]': finished 2024-03-16T14:29:41.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:41 smithi151 ceph-mon[30603]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T14:29:41.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:41 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:42.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/4112262032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]: dispatch 2024-03-16T14:29:42.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]: dispatch 2024-03-16T14:29:42.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]': finished 2024-03-16T14:29:42.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[25218]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T14:29:42.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:42.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/4112262032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]: dispatch 2024-03-16T14:29:42.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]: dispatch 2024-03-16T14:29:42.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffea97ec-5a5f-4dc9-a5ba-a7d304280675"}]': finished 2024-03-16T14:29:42.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[38287]: osdmap e35: 6 total, 5 up, 6 in 2024-03-16T14:29:42.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:41 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:42.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:42 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/1233031045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:29:42.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:42 smithi151 ceph-mon[30603]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-16T14:29:43.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:42 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/1233031045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:29:43.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:42 smithi026 ceph-mon[25218]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-16T14:29:43.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:42 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/1233031045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:29:43.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:42 smithi026 ceph-mon[38287]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-16T14:29:44.902 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:44 smithi151 ceph-mon[30603]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T14:29:45.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:44 smithi026 ceph-mon[38287]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T14:29:45.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:44 smithi026 ceph-mon[25218]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T14:29:46.928 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:46 smithi151 ceph-mon[30603]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T14:29:47.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:46 smithi026 ceph-mon[38287]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T14:29:47.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:46 smithi026 ceph-mon[25218]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-03-16T14:29:47.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:47 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T14:29:47.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:47 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:48.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:47 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T14:29:48.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:47 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:48.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:47 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-16T14:29:48.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:47 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:48.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:48 smithi151 ceph-mon[30603]: Deploying daemon osd.5 on smithi151 2024-03-16T14:29:48.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:48 smithi151 ceph-mon[30603]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T14:29:49.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:48 smithi026 ceph-mon[25218]: Deploying daemon osd.5 on smithi151 2024-03-16T14:29:49.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:48 smithi026 ceph-mon[25218]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T14:29:49.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:48 smithi026 ceph-mon[38287]: Deploying daemon osd.5 on smithi151 2024-03-16T14:29:49.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:48 smithi026 ceph-mon[38287]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T14:29:50.978 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:50 smithi151 ceph-mon[30603]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:51.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:50 smithi026 ceph-mon[25218]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:51.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:50 smithi026 ceph-mon[38287]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:52.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:52 smithi151 ceph-mon[30603]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:53.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:52 smithi026 ceph-mon[25218]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:53.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:52 smithi026 ceph-mon[38287]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:54.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:54 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:54.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:54 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:54.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:54 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:54.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:54 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:54.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:54 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:54.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:54 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:54.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:54 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:54.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:54 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:54.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:54 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:55.615 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:55 smithi151 ceph-mon[30603]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:55.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:55 smithi026 ceph-mon[25218]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:55.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:55 smithi026 ceph-mon[38287]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:57.274 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 5 on host 'smithi151' 2024-03-16T14:29:57.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:57.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:57.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: from='osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T14:29:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T14:29:57.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:57 smithi151 ceph-mon[30603]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:57.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:57.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:57.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: from='osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T14:29:57.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[38287]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: from='osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-16T14:29:57.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:57 smithi026 ceph-mon[25218]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:57.821 DEBUG:teuthology.orchestra.run.smithi151:osd.5> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.5.service 2024-03-16T14:29:57.823 INFO:tasks.cephadm:Deploying osd.6 on smithi151 with /dev/nvme2n1... 2024-03-16T14:29:57.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-16T14:29:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T14:29:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: from='osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:58.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T14:29:58.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:58.488 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-mon[30603]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:58.488 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:29:58.240+0000 7f8c2cc0d700 -1 osd.5 0 waiting for initial osdmap 2024-03-16T14:29:58.488 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:29:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:29:58.245+0000 7f8c28a00700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:29:58.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:58.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:58.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:58.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T14:29:58.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: from='osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[25218]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: from='osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: osdmap e36: 6 total, 5 up, 6 in 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:58.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:58 smithi026 ceph-mon[38287]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:29:58.589 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:29:59.454 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:29:59.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T14:29:59.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:59.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105] boot 2024-03-16T14:29:59.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T14:29:59.455 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:29:59 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105] boot 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T14:29:59.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: osdmap e37: 6 total, 5 up, 6 in 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: osd.5 [v2:172.21.15.151:6808/7047105,v1:172.21.15.151:6809/7047105] boot 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: osdmap e38: 6 total, 6 up, 6 in 2024-03-16T14:29:59.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:29:59 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:30:00.385 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:00 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:30:00.386 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:00 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:30:00.386 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:00 smithi151 ceph-mon[30603]: overall HEALTH_OK 2024-03-16T14:30:00.386 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:00 smithi151 ceph-mon[30603]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T14:30:00.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:30:00.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:30:00.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[25218]: overall HEALTH_OK 2024-03-16T14:30:00.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[25218]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T14:30:00.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:30:00.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:30:00.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[38287]: overall HEALTH_OK 2024-03-16T14:30:00.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:00 smithi026 ceph-mon[38287]: osdmap e39: 6 total, 6 up, 6 in 2024-03-16T14:30:01.400 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:01 smithi151 ceph-mon[30603]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:01.400 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:01 smithi151 ceph-mon[30603]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T14:30:01.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:01 smithi026 ceph-mon[38287]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:01.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:01 smithi026 ceph-mon[38287]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T14:30:01.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:01 smithi026 ceph-mon[25218]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:01.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:01 smithi026 ceph-mon[25218]: osdmap e40: 6 total, 6 up, 6 in 2024-03-16T14:30:02.002 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/nvme2n1 2024-03-16T14:30:02.002 INFO:teuthology.orchestra.run.smithi151.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:30:02.002 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-16T14:30:02.002 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-16T14:30:02.003 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-16T14:30:02.003 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137568 s, 762 MB/s 2024-03-16T14:30:02.003 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-16T14:30:02.688 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi151 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi151 to 8544M 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:02.689 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:02 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi151:/dev/nvme2n1 2024-03-16T14:30:02.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi151 2024-03-16T14:30:02.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:02.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi151 to 8544M 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi151 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:02.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi151 to 8544M 2024-03-16T14:30:02.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:02.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:02.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:02 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:02.949 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:30:03.719 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:03 smithi151 ceph-mon[30603]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T14:30:03.720 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:03 smithi151 ceph-mon[30603]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:03.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:03 smithi026 ceph-mon[25218]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T14:30:03.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:03 smithi026 ceph-mon[25218]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:03.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:03 smithi026 ceph-mon[38287]: osdmap e41: 6 total, 6 up, 6 in 2024-03-16T14:30:03.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:03 smithi026 ceph-mon[38287]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:04.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:04 smithi151 ceph-mon[30603]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T14:30:04.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:04 smithi026 ceph-mon[25218]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T14:30:04.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:04 smithi026 ceph-mon[38287]: osdmap e42: 6 total, 6 up, 6 in 2024-03-16T14:30:05.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:05 smithi151 ceph-mon[30603]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-16T14:30:05.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:30:05.549 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:30:05.550 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:05.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[25218]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-16T14:30:05.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:30:05.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:30:05.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:05.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[38287]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-16T14:30:05.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:30:05.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:30:05.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:06.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:06 smithi151 ceph-mon[30603]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:30:06.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:06 smithi026 ceph-mon[25218]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:30:06.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:06 smithi026 ceph-mon[38287]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:30:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[25218]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:30:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/1523997414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]: dispatch 2024-03-16T14:30:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]: dispatch 2024-03-16T14:30:07.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]': finished 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[25218]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[38287]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/1523997414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]: dispatch 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]: dispatch 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]': finished 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[38287]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T14:30:07.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:07 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:07 smithi151 ceph-mon[30603]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:30:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:07 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/1523997414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]: dispatch 2024-03-16T14:30:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:07 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]: dispatch 2024-03-16T14:30:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:07 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "640fca26-3089-4b9a-9efd-0a054f336457"}]': finished 2024-03-16T14:30:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:07 smithi151 ceph-mon[30603]: osdmap e43: 7 total, 6 up, 7 in 2024-03-16T14:30:07.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:07 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:08.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:08 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/2552220875' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:30:08.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:08 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/2552220875' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:30:08.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:08 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/2552220875' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:30:09.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:09 smithi026 ceph-mon[25218]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-16T14:30:09.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:09 smithi026 ceph-mon[38287]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-16T14:30:09.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:09 smithi151 ceph-mon[30603]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-03-16T14:30:10.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:10 smithi151 ceph-mon[30603]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T14:30:11.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:10 smithi026 ceph-mon[25218]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T14:30:11.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:10 smithi026 ceph-mon[38287]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-16T14:30:12.885 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:12 smithi151 ceph-mon[30603]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-16T14:30:13.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:12 smithi026 ceph-mon[25218]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-16T14:30:13.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:12 smithi026 ceph-mon[38287]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-16T14:30:14.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:14 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T14:30:14.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:14 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:14.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:14 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T14:30:14.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:14 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:14.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-16T14:30:14.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:14 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:15.439 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:15 smithi151 ceph-mon[30603]: Deploying daemon osd.6 on smithi151 2024-03-16T14:30:15.439 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:15 smithi151 ceph-mon[30603]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:15.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:15 smithi026 ceph-mon[25218]: Deploying daemon osd.6 on smithi151 2024-03-16T14:30:15.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:15 smithi026 ceph-mon[25218]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:15.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:15 smithi026 ceph-mon[38287]: Deploying daemon osd.6 on smithi151 2024-03-16T14:30:15.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:15 smithi026 ceph-mon[38287]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:16.953 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:16 smithi151 ceph-mon[30603]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:17.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:16 smithi026 ceph-mon[25218]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:17.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:16 smithi026 ceph-mon[38287]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:18.844 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:18 smithi151 ceph-mon[30603]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:19.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:18 smithi026 ceph-mon[25218]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:19.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:18 smithi026 ceph-mon[38287]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:20.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:20 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:20.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:20 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:20.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:20 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:20.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:20 smithi151 ceph-mon[30603]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[25218]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:21.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:21.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:21.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:21.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:20 smithi026 ceph-mon[38287]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:22.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:22 smithi151 ceph-mon[30603]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:22.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:22.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:22.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:22.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:22.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[25218]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:23.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:23.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[38287]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:23.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:23.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:23.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.623 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 6 on host 'smithi151' 2024-03-16T14:30:23.958 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:23 smithi151 ceph-mon[30603]: from='osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T14:30:23.958 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:23 smithi151 ceph-mon[30603]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T14:30:23.958 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:23 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.958 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:23 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:23.959 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:23 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:24.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[25218]: from='osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T14:30:24.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[25218]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T14:30:24.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:24.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:24.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:24.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[38287]: from='osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T14:30:24.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[38287]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-16T14:30:24.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:24.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:24.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:23 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:24.204 DEBUG:teuthology.orchestra.run.smithi151:osd.6> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.6.service 2024-03-16T14:30:24.206 INFO:tasks.cephadm:Deploying osd.7 on smithi151 with /dev/nvme1n1... 2024-03-16T14:30:24.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-16T14:30:24.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:30:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-mon[30603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T14:30:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-mon[30603]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T14:30:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-mon[30603]: from='osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-mon[30603]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-mon[30603]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:24.988 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:30:24.663+0000 7f16aa11e700 -1 osd.6 0 waiting for initial osdmap 2024-03-16T14:30:24.988 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:30:24 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:30:24.671+0000 7f16a5f11700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:30:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[25218]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T14:30:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[25218]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T14:30:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[25218]: from='osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[25218]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[25218]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:25.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[38287]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-16T14:30:25.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[38287]: osdmap e44: 7 total, 6 up, 7 in 2024-03-16T14:30:25.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:25.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[38287]: from='osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:25.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[38287]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:25.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:24 smithi026 ceph-mon[38287]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-16T14:30:25.867 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:25 smithi151 ceph-mon[30603]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:30:25.867 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:25 smithi151 ceph-mon[30603]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T14:30:25.867 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:25 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:25.868 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:25 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:26.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[25218]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:30:26.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[25218]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T14:30:26.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:26.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:26.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[38287]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:30:26.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[38287]: osdmap e45: 7 total, 6 up, 7 in 2024-03-16T14:30:26.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:26.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:25 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:26.857 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:26 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:30:26.857 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:26 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:30:26.857 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:26 smithi151 ceph-mon[30603]: osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676] boot 2024-03-16T14:30:26.857 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:26 smithi151 ceph-mon[30603]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T14:30:26.857 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:26 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:26.857 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:26 smithi151 ceph-mon[30603]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:27.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:30:27.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:30:27.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[38287]: osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676] boot 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[38287]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[38287]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[25218]: osd.6 [v2:172.21.15.151:6816/2840379676,v1:172.21.15.151:6817/2840379676] boot 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[25218]: osdmap e46: 7 total, 7 up, 7 in 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:30:27.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:26 smithi026 ceph-mon[25218]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:27.134 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping: /dev/nvme1n1 2024-03-16T14:30:27.134 INFO:teuthology.orchestra.run.smithi151.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-16T14:30:27.134 INFO:teuthology.orchestra.run.smithi151.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-16T14:30:27.134 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10+0 records in 2024-03-16T14:30:27.134 INFO:teuthology.orchestra.run.smithi151.stderr:10+0 records out 2024-03-16T14:30:27.134 INFO:teuthology.orchestra.run.smithi151.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132484 s, 791 MB/s 2024-03-16T14:30:27.134 INFO:teuthology.orchestra.run.smithi151.stderr:--> Zapping successful for: 2024-03-16T14:30:27.843 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:27 smithi151 ceph-mon[30603]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T14:30:27.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi151:/dev/nvme1n1 2024-03-16T14:30:28.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:27 smithi026 ceph-mon[25218]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T14:30:28.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:27 smithi026 ceph-mon[38287]: osdmap e47: 7 total, 7 up, 7 in 2024-03-16T14:30:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:30:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:28 smithi151 ceph-mon[30603]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T14:30:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:28 smithi151 ceph-mon[30603]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:29.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:28 smithi026 ceph-mon[25218]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T14:30:29.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:28 smithi026 ceph-mon[25218]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:29.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:28 smithi026 ceph-mon[38287]: osdmap e48: 7 total, 7 up, 7 in 2024-03-16T14:30:29.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:28 smithi026 ceph-mon[38287]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: osdmap e49: 7 total, 7 up, 7 in 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi151 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi151 to 5696M 2024-03-16T14:30:29.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:29.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:29.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:29.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:29 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: osdmap e49: 7 total, 7 up, 7 in 2024-03-16T14:30:30.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi151 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi151 to 5696M 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: osdmap e49: 7 total, 7 up, 7 in 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi151 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi151 to 5696M 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:30.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:29 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:30 smithi151 ceph-mon[30603]: osdmap e50: 7 total, 7 up, 7 in 2024-03-16T14:30:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:30 smithi151 ceph-mon[30603]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:30:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:30:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[25218]: osdmap e50: 7 total, 7 up, 7 in 2024-03-16T14:30:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[25218]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:30:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:30:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:31.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[38287]: osdmap e50: 7 total, 7 up, 7 in 2024-03-16T14:30:31.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[38287]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:31.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-16T14:30:31.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-16T14:30:31.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:31.914 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:31 smithi151 ceph-mon[30603]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:30:32.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:31 smithi026 ceph-mon[25218]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:30:32.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:31 smithi026 ceph-mon[38287]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:30:32.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:32 smithi151 ceph-mon[30603]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-16T14:30:33.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:32 smithi026 ceph-mon[25218]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-16T14:30:33.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:32 smithi026 ceph-mon[38287]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-16T14:30:33.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:33 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/1290435300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]: dispatch 2024-03-16T14:30:33.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:33 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]: dispatch 2024-03-16T14:30:33.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:33 smithi151 ceph-mon[30603]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]': finished 2024-03-16T14:30:33.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:33 smithi151 ceph-mon[30603]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T14:30:33.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:33 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/1290435300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]: dispatch 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]: dispatch 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[25218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]': finished 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[25218]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/1290435300' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]: dispatch 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]: dispatch 2024-03-16T14:30:34.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[38287]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80b545b9-c39f-4330-9c35-82e904945f61"}]': finished 2024-03-16T14:30:34.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[38287]: osdmap e51: 8 total, 7 up, 8 in 2024-03-16T14:30:34.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:33 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:34.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:34 smithi151 ceph-mon[30603]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:30:34.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:34 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/1629937893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:30:35.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:34 smithi026 ceph-mon[25218]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:30:35.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:34 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/1629937893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:30:35.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:34 smithi026 ceph-mon[38287]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-16T14:30:35.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:34 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/1629937893' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-16T14:30:36.899 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:36 smithi151 ceph-mon[30603]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-16T14:30:37.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:36 smithi026 ceph-mon[25218]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-16T14:30:37.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:36 smithi026 ceph-mon[38287]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-16T14:30:38.900 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:38 smithi151 ceph-mon[30603]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-16T14:30:39.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:38 smithi026 ceph-mon[25218]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-16T14:30:39.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:38 smithi026 ceph-mon[38287]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-16T14:30:40.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:40 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T14:30:40.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:40 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:40.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:40 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T14:30:40.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:40 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:40.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:40 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-16T14:30:40.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:40 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:41.459 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:41 smithi151 ceph-mon[30603]: Deploying daemon osd.7 on smithi151 2024-03-16T14:30:41.459 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:41 smithi151 ceph-mon[30603]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T14:30:41.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:41 smithi026 ceph-mon[25218]: Deploying daemon osd.7 on smithi151 2024-03-16T14:30:41.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:41 smithi026 ceph-mon[25218]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T14:30:41.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:41 smithi026 ceph-mon[38287]: Deploying daemon osd.7 on smithi151 2024-03-16T14:30:41.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:41 smithi026 ceph-mon[38287]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-16T14:30:42.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:42 smithi151 ceph-mon[30603]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:43.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:42 smithi026 ceph-mon[25218]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:43.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:42 smithi026 ceph-mon[38287]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:44.959 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:44 smithi151 ceph-mon[30603]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:45.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:44 smithi026 ceph-mon[25218]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:45.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:44 smithi026 ceph-mon[38287]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:47.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:46 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:47.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:46 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:47.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:46 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:47.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:46 smithi151 ceph-mon[30603]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:47.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:47.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:47.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:47.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[25218]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:47.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:47.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:47.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:47.256 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:46 smithi026 ceph-mon[38287]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:48.916 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:48 smithi151 ceph-mon[30603]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:49.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:48 smithi026 ceph-mon[38287]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:49.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:48 smithi026 ceph-mon[25218]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:49.164 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 7 on host 'smithi151' 2024-03-16T14:30:49.774 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:49 smithi151 ceph-mon[30603]: from='osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T14:30:49.775 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:49 smithi151 ceph-mon[30603]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T14:30:49.775 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:49.775 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:49 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:49.776 DEBUG:teuthology.orchestra.run.smithi151:osd.7> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.7.service 2024-03-16T14:30:49.778 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-16T14:30:49.778 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-16T14:30:50.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[38287]: from='osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T14:30:50.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[38287]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T14:30:50.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[25218]: from='osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T14:30:50.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[25218]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-16T14:30:50.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:49 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.042 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:30:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T14:30:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: osdmap e52: 8 total, 7 up, 8 in 2024-03-16T14:30:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: from='osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[38287]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: osdmap e52: 8 total, 7 up, 8 in 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: from='osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:50 smithi026 ceph-mon[25218]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: osdmap e52: 8 total, 7 up, 8 in 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: from='osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]: dispatch 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:50.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:50 smithi151 ceph-mon[30603]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:51.453 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:30:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:30:51.132+0000 7fbe9e420700 -1 osd.7 0 waiting for initial osdmap 2024-03-16T14:30:51.454 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:30:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:30:51.139+0000 7fbe9620b700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-16T14:30:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:51 smithi151 ceph-mon[30603]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:30:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:51 smithi151 ceph-mon[30603]: osdmap e53: 8 total, 7 up, 8 in 2024-03-16T14:30:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:51 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:51 smithi151 ceph-mon[30603]: from='osd.7 ' entity='osd.7' 2024-03-16T14:30:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[25218]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:30:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[25218]: osdmap e53: 8 total, 7 up, 8 in 2024-03-16T14:30:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[25218]: from='osd.7 ' entity='osd.7' 2024-03-16T14:30:51.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[38287]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-03-16T14:30:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[38287]: osdmap e53: 8 total, 7 up, 8 in 2024-03-16T14:30:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:51 smithi026 ceph-mon[38287]: from='osd.7 ' entity='osd.7' 2024-03-16T14:30:51.902 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:30:52.609 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1710599451,"num_in_osds":8,"osd_in_since":1710599433,"num_remapped_pgs":1} 2024-03-16T14:30:52.610 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T14:30:52.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: purged_snaps scrub starts 2024-03-16T14:30:52.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: purged_snaps scrub ok 2024-03-16T14:30:52.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:52.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800] boot 2024-03-16T14:30:52.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T14:30:52.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:52.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3999365268' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T14:30:52.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:52 smithi151 ceph-mon[30603]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:52.868 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: purged_snaps scrub starts 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: purged_snaps scrub ok 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800] boot 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3999365268' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T14:30:52.991 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[38287]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: purged_snaps scrub starts 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: purged_snaps scrub ok 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: osd.7 [v2:172.21.15.151:6824/2379156800,v1:172.21.15.151:6825/2379156800] boot 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: osdmap e54: 8 total, 8 up, 8 in 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3999365268' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-16T14:30:52.992 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:52 smithi026 ceph-mon[25218]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-16T14:30:53.865 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:53 smithi026 ceph-mon[38287]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T14:30:53.865 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:53 smithi026 ceph-mon[25218]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T14:30:53.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:53 smithi151 ceph-mon[30603]: osdmap e55: 8 total, 8 up, 8 in 2024-03-16T14:30:54.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:54 smithi026 ceph-mon[38287]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T14:30:54.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:54 smithi026 ceph-mon[38287]: osdmap e57: 8 total, 8 up, 8 in 2024-03-16T14:30:54.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:54 smithi026 ceph-mon[38287]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-16T14:30:54.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:54 smithi026 ceph-mon[25218]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T14:30:54.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:54 smithi026 ceph-mon[25218]: osdmap e57: 8 total, 8 up, 8 in 2024-03-16T14:30:54.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:54 smithi026 ceph-mon[25218]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-16T14:30:54.819 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:30:54.820 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":57,"fsid":"efcc9110-e3a0-11ee-95c9-87774f69a715","created":"2024-03-16T14:25:30.002920+0000","modified":"2024-03-16T14:30:54.225427+0000","last_up_change":"2024-03-16T14:30:51.676817+0000","last_in_change":"2024-03-16T14:30:33.385351+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T14:28:44.874373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e773bb90-a94a-471e-8d8b-51a7cbada97b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6803","nonce":2042975011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6805","nonce":2042975011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6809","nonce":2042975011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6807","nonce":2042975011}]},"public_addr":"172.21.15.26:6803/2042975011","cluster_addr":"172.21.15.26:6805/2042975011","heartbeat_back_addr":"172.21.15.26:6809/2042975011","heartbeat_front_addr":"172.21.15.26:6807/2042975011","state":["exists","up"]},{"osd":1,"uuid":"79b74b66-d263-4eb1-af4c-76fcef030e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6811","nonce":2687555532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6813","nonce":2687555532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6817","nonce":2687555532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6815","nonce":2687555532}]},"public_addr":"172.21.15.26:6811/2687555532","cluster_addr":"172.21.15.26:6813/2687555532","heartbeat_back_addr":"172.21.15.26:6817/2687555532","heartbeat_front_addr":"172.21.15.26:6815/2687555532","state":["exists","up"]},{"osd":2,"uuid":"7a33de3b-ab23-4c5f-aeef-5688c34717db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6819","nonce":4025043318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6821","nonce":4025043318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6825","nonce":4025043318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6823","nonce":4025043318}]},"public_addr":"172.21.15.26:6819/4025043318","cluster_addr":"172.21.15.26:6821/4025043318","heartbeat_back_addr":"172.21.15.26:6825/4025043318","heartbeat_front_addr":"172.21.15.26:6823/4025043318","state":["exists","up"]},{"osd":3,"uuid":"b12f1441-6a57-485e-8bbc-7ae868ed9d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6827","nonce":3222197319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6829","nonce":3222197319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6833","nonce":3222197319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6831","nonce":3222197319}]},"public_addr":"172.21.15.26:6827/3222197319","cluster_addr":"172.21.15.26:6829/3222197319","heartbeat_back_addr":"172.21.15.26:6833/3222197319","heartbeat_front_addr":"172.21.15.26:6831/3222197319","state":["exists","up"]},{"osd":4,"uuid":"9abf790f-4b0c-4f91-893b-356b795f985d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6801","nonce":1799636760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6803","nonce":1799636760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6807","nonce":1799636760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6805","nonce":1799636760}]},"public_addr":"172.21.15.151:6801/1799636760","cluster_addr":"172.21.15.151:6803/1799636760","heartbeat_back_addr":"172.21.15.151:6807/1799636760","heartbeat_front_addr":"172.21.15.151:6805/1799636760","state":["exists","up"]},{"osd":5,"uuid":"ffea97ec-5a5f-4dc9-a5ba-a7d304280675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6809","nonce":7047105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6811","nonce":7047105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6815","nonce":7047105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6813","nonce":7047105}]},"public_addr":"172.21.15.151:6809/7047105","cluster_addr":"172.21.15.151:6811/7047105","heartbeat_back_addr":"172.21.15.151:6815/7047105","heartbeat_front_addr":"172.21.15.151:6813/7047105","state":["exists","up"]},{"osd":6,"uuid":"640fca26-3089-4b9a-9efd-0a054f336457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6817","nonce":2840379676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6819","nonce":2840379676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6823","nonce":2840379676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6821","nonce":2840379676}]},"public_addr":"172.21.15.151:6817/2840379676","cluster_addr":"172.21.15.151:6819/2840379676","heartbeat_back_addr":"172.21.15.151:6823/2840379676","heartbeat_front_addr":"172.21.15.151:6821/2840379676","state":["exists","up"]},{"osd":7,"uuid":"80b545b9-c39f-4330-9c35-82e904945f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6825","nonce":2379156800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6827","nonce":2379156800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6831","nonce":2379156800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6829","nonce":2379156800}]},"public_addr":"172.21.15.151:6825/2379156800","cluster_addr":"172.21.15.151:6827/2379156800","heartbeat_back_addr":"172.21.15.151:6831/2379156800","heartbeat_front_addr":"172.21.15.151:6829/2379156800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:27:51.608819+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:28:16.714427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:28:41.938355+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:07.808874+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:31.573352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:57.599378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:30:23.859894+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:30:49.882000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/929495780":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/1583238033":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/3218505860":"2024-03-17T14:26:30.528857+0000","172.21.15.26:6801/1667601179":"2024-03-17T14:26:04.256747+0000","172.21.15.26:6800/1667601179":"2024-03-17T14:26:04.256747+0000","172.21.15.26:6800/929495780":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/1857687384":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/659101530":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/3734138219":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/3185583976":"2024-03-17T14:26:30.528857+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T14:30:54.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:54 smithi151 ceph-mon[30603]: osdmap e56: 8 total, 8 up, 8 in 2024-03-16T14:30:54.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:54 smithi151 ceph-mon[30603]: osdmap e57: 8 total, 8 up, 8 in 2024-03-16T14:30:54.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:54 smithi151 ceph-mon[30603]: pgmap v179: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-16T14:30:55.540 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-16T14:28:44.874373+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-16T14:30:55.540 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-16T14:30:55.783 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: Detected new or changed devices on smithi151 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.904 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi151 to 4272M 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1502551767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[25218]: osdmap e58: 8 total, 8 up, 8 in 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: Detected new or changed devices on smithi151 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.905 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi151 to 4272M 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1502551767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.906 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:55 smithi026 ceph-mon[38287]: osdmap e58: 8 total, 8 up, 8 in 2024-03-16T14:30:55.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: Detected new or changed devices on smithi151 2024-03-16T14:30:55.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi151 to 4272M 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1502551767' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:30:55.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:55 smithi151 ceph-mon[30603]: osdmap e58: 8 total, 8 up, 8 in 2024-03-16T14:30:56.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:56 smithi151 ceph-mon[30603]: pgmap v181: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T14:30:57.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:56 smithi026 ceph-mon[25218]: pgmap v181: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T14:30:57.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:56 smithi026 ceph-mon[38287]: pgmap v181: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-16T14:30:57.591 INFO:teuthology.orchestra.run.smithi026.stdout:pg_num: 1 2024-03-16T14:30:57.951 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:57 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3616954609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T14:30:57.951 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:57 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3616954609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T14:30:57.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:57 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3616954609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-16T14:30:58.239 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi026 2024-03-16T14:30:58.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi026=foo.a' 2024-03-16T14:30:58.492 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:30:58.866 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:30:58 smithi151 ceph-mon[30603]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-16T14:30:59.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:30:58 smithi026 ceph-mon[25218]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-16T14:30:59.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:30:58 smithi026 ceph-mon[38287]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-16T14:31:00.312 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled rgw.foo.a update... 2024-03-16T14:31:00.871 DEBUG:teuthology.orchestra.run.smithi026:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@rgw.foo.a.service 2024-03-16T14:31:00.873 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi151 2024-03-16T14:31:00.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-16T14:31:01.112 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:01.484 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi026=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:01.484 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: Saving service rgw.foo.a spec with placement smithi026=foo.a;count:1 2024-03-16T14:31:01.484 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.484 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:01.484 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:01.484 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:01.484 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.485 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-16T14:31:01.485 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-16T14:31:01.485 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.485 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:01.485 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: Deploying daemon rgw.foo.a on smithi026 2024-03-16T14:31:01.485 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:01 smithi151 ceph-mon[30603]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-16T14:31:01.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi026=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:01.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: Saving service rgw.foo.a spec with placement smithi026=foo.a;count:1 2024-03-16T14:31:01.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: Deploying daemon rgw.foo.a on smithi026 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[25218]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi026=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: Saving service rgw.foo.a spec with placement smithi026=foo.a;count:1 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: Deploying daemon rgw.foo.a on smithi026 2024-03-16T14:31:01.507 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:01 smithi026 ceph-mon[38287]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-16T14:31:02.505 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:31:02 smithi026 systemd[1]: Starting Ceph rgw.foo.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:31:02.895 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:02 smithi026 ceph-mon[38287]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:02.895 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:02 smithi026 ceph-mon[25218]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:02.895 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:31:02 smithi026 podman[73579]: 2024-03-16 14:31:02.527433749 +0000 UTC m=+0.017466769 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:31:02.896 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:31:02 smithi026 podman[73579]: 2024-03-16 14:31:02.641681768 +0000 UTC m=+0.131714776 container create 3c2ac519e932419586a71b93bfe62b595ad7d854109537a4625a1fd8779cad4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-16T14:31:02.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:02 smithi151 ceph-mon[30603]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:03.255 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:31:02 smithi026 podman[73579]: 2024-03-16 14:31:02.893938419 +0000 UTC m=+0.383971427 container init 3c2ac519e932419586a71b93bfe62b595ad7d854109537a4625a1fd8779cad4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-16T14:31:03.255 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:31:02 smithi026 podman[73579]: 2024-03-16 14:31:02.896710069 +0000 UTC m=+0.386743082 container start 3c2ac519e932419586a71b93bfe62b595ad7d854109537a4625a1fd8779cad4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-16T14:31:03.255 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:31:02 smithi026 bash[73579]: 3c2ac519e932419586a71b93bfe62b595ad7d854109537a4625a1fd8779cad4b 2024-03-16T14:31:03.255 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:31:02 smithi026 systemd[1]: Started Ceph rgw.foo.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:31:03.668 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'datapool' created 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/4119682323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: Saving service rgw.foo.a spec with placement smithi026=foo.a;count:1 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:03.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:03.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:03 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:04.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/4119682323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-16T14:31:04.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-16T14:31:04.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: Saving service rgw.foo.a spec with placement smithi026=foo.a;count:1 2024-03-16T14:31:04.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/4119682323' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: Saving service rgw.foo.a spec with placement smithi026=foo.a;count:1 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:04.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:03 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:04.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- rbd pool init datapool 2024-03-16T14:31:04.599 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:04 smithi151 ceph-mon[30603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-16T14:31:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:04 smithi151 ceph-mon[30603]: osdmap e59: 8 total, 8 up, 8 in 2024-03-16T14:31:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:04 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/2918077853' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-16T14:31:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:04 smithi151 ceph-mon[30603]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-16T14:31:04.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:04 smithi151 ceph-mon[30603]: pgmap v186: 36 pgs: 20 creating+peering, 1 active+clean, 15 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:05.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[38287]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-16T14:31:05.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[38287]: osdmap e59: 8 total, 8 up, 8 in 2024-03-16T14:31:05.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/2918077853' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-16T14:31:05.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[38287]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-16T14:31:05.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[38287]: pgmap v186: 36 pgs: 20 creating+peering, 1 active+clean, 15 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:05.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[25218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-16T14:31:05.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[25218]: osdmap e59: 8 total, 8 up, 8 in 2024-03-16T14:31:05.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2918077853' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-16T14:31:05.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[25218]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-16T14:31:05.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:04 smithi026 ceph-mon[25218]: pgmap v186: 36 pgs: 20 creating+peering, 1 active+clean, 15 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:05.868 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: osdmap e60: 8 total, 8 up, 8 in 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: Checking dashboard <-> RGW credentials 2024-03-16T14:31:05.869 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:05 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:06.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T14:31:06.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-16T14:31:06.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: osdmap e60: 8 total, 8 up, 8 in 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: Checking dashboard <-> RGW credentials 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: osdmap e60: 8 total, 8 up, 8 in 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:06.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:06.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:06.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:06.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: Checking dashboard <-> RGW credentials 2024-03-16T14:31:06.007 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:05 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:06.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:06 smithi151 ceph-mon[30603]: osdmap e61: 8 total, 8 up, 8 in 2024-03-16T14:31:06.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:06 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-16T14:31:06.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:06 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-16T14:31:06.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:06 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/4028281302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-16T14:31:06.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:06 smithi151 ceph-mon[30603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-16T14:31:06.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:06 smithi151 ceph-mon[30603]: pgmap v189: 68 pgs: 20 creating+peering, 9 active+clean, 39 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[38287]: osdmap e61: 8 total, 8 up, 8 in 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/4028281302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[38287]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[38287]: pgmap v189: 68 pgs: 20 creating+peering, 9 active+clean, 39 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[25218]: osdmap e61: 8 total, 8 up, 8 in 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-16T14:31:07.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-16T14:31:07.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/4028281302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-16T14:31:07.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[25218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-16T14:31:07.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:06 smithi026 ceph-mon[25218]: pgmap v189: 68 pgs: 20 creating+peering, 9 active+clean, 39 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:07 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-16T14:31:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:07 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-16T14:31:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:07 smithi151 ceph-mon[30603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-16T14:31:07.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:07 smithi151 ceph-mon[30603]: osdmap e62: 8 total, 8 up, 8 in 2024-03-16T14:31:08.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-16T14:31:08.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-16T14:31:08.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[25218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-16T14:31:08.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[25218]: osdmap e62: 8 total, 8 up, 8 in 2024-03-16T14:31:08.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-16T14:31:08.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-16T14:31:08.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[38287]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-16T14:31:08.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:07 smithi026 ceph-mon[38287]: osdmap e62: 8 total, 8 up, 8 in 2024-03-16T14:31:08.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:08 smithi151 ceph-mon[30603]: osdmap e63: 8 total, 8 up, 8 in 2024-03-16T14:31:08.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:08 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-16T14:31:08.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:08 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-16T14:31:08.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:08 smithi151 ceph-mon[30603]: pgmap v192: 100 pgs: 20 creating+peering, 16 active+clean, 64 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[25218]: osdmap e63: 8 total, 8 up, 8 in 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[25218]: pgmap v192: 100 pgs: 20 creating+peering, 16 active+clean, 64 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[38287]: osdmap e63: 8 total, 8 up, 8 in 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-16T14:31:09.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:08 smithi026 ceph-mon[38287]: pgmap v192: 100 pgs: 20 creating+peering, 16 active+clean, 64 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-16T14:31:09.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.151 --placement '1;smithi151=iscsi.a' 2024-03-16T14:31:09.565 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:09.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:09 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-16T14:31:09.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:09 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-16T14:31:09.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:09 smithi151 ceph-mon[30603]: osdmap e64: 8 total, 8 up, 8 in 2024-03-16T14:31:10.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:09 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-16T14:31:10.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:09 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-16T14:31:10.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:09 smithi026 ceph-mon[25218]: osdmap e64: 8 total, 8 up, 8 in 2024-03-16T14:31:10.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:09 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-16T14:31:10.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:09 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-16T14:31:10.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:09 smithi026 ceph-mon[38287]: osdmap e64: 8 total, 8 up, 8 in 2024-03-16T14:31:10.842 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:10 smithi151 ceph-mon[30603]: osdmap e65: 8 total, 8 up, 8 in 2024-03-16T14:31:10.842 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:10 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-16T14:31:10.842 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:10 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-16T14:31:10.842 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:10 smithi151 ceph-mon[30603]: pgmap v195: 132 pgs: 7 creating+peering, 90 active+clean, 35 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T14:31:11.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[25218]: osdmap e65: 8 total, 8 up, 8 in 2024-03-16T14:31:11.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-16T14:31:11.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-16T14:31:11.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[25218]: pgmap v195: 132 pgs: 7 creating+peering, 90 active+clean, 35 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T14:31:11.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[38287]: osdmap e65: 8 total, 8 up, 8 in 2024-03-16T14:31:11.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-16T14:31:11.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-16T14:31:11.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:10 smithi026 ceph-mon[38287]: pgmap v195: 132 pgs: 7 creating+peering, 90 active+clean, 35 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-16T14:31:11.282 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled iscsi.datapool update... 2024-03-16T14:31:11.908 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-16T14:31:11.908 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-16T14:31:11.908 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: osdmap e66: 8 total, 8 up, 8 in 2024-03-16T14:31:11.908 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-16T14:31:11.908 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-16T14:31:11.909 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.151", "placement": "1;smithi151=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:11.909 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: Saving service iscsi.datapool spec with placement smithi151=iscsi.a;count:1 2024-03-16T14:31:11.909 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:11 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:11.940 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-16T14:31:11.940 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:31:11.941 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-16T14:31:11.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-16T14:31:11.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-16T14:31:11.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: osdmap e66: 8 total, 8 up, 8 in 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.151", "placement": "1;smithi151=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: Saving service iscsi.datapool spec with placement smithi151=iscsi.a;count:1 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: osdmap e66: 8 total, 8 up, 8 in 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-16T14:31:11.971 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.151", "placement": "1;smithi151=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: Saving service iscsi.datapool spec with placement smithi151=iscsi.a;count:1 2024-03-16T14:31:11.972 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:11 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:11.976 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:31:11.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-16T14:31:12.009 DEBUG:teuthology.orchestra.run.smithi151:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@iscsi.iscsi.a.service 2024-03-16T14:31:12.052 INFO:tasks.cephadm:Adding prometheus.a on smithi151 2024-03-16T14:31:12.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply prometheus '1;smithi151=a' 2024-03-16T14:31:12.352 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: osdmap e67: 8 total, 8 up, 8 in 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-16T14:31:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-16T14:31:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: Deploying daemon iscsi.iscsi.a on smithi151 2024-03-16T14:31:12.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:12 smithi151 ceph-mon[30603]: pgmap v198: 132 pgs: 7 creating+peering, 96 active+clean, 29 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-16T14:31:13.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-16T14:31:13.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-16T14:31:13.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: osdmap e67: 8 total, 8 up, 8 in 2024-03-16T14:31:13.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: Deploying daemon iscsi.iscsi.a on smithi151 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[25218]: pgmap v198: 132 pgs: 7 creating+peering, 96 active+clean, 29 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1305836412' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4104995880' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: osdmap e67: 8 total, 8 up, 8 in 2024-03-16T14:31:13.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:13.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:13.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:13.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-16T14:31:13.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-16T14:31:13.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:13.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: Deploying daemon iscsi.iscsi.a on smithi151 2024-03-16T14:31:13.007 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:12 smithi026 ceph-mon[38287]: pgmap v198: 132 pgs: 7 creating+peering, 96 active+clean, 29 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-16T14:31:13.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:13 smithi151 ceph-mon[30603]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T14:31:13.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:13 smithi151 ceph-mon[30603]: Cluster is now healthy 2024-03-16T14:31:14.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:13 smithi026 ceph-mon[25218]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T14:31:14.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:13 smithi026 ceph-mon[25218]: Cluster is now healthy 2024-03-16T14:31:14.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:13 smithi026 ceph-mon[38287]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-16T14:31:14.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:13 smithi026 ceph-mon[38287]: Cluster is now healthy 2024-03-16T14:31:14.192 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled prometheus update... 2024-03-16T14:31:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:15 smithi151 ceph-mon[30603]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:15 smithi151 ceph-mon[30603]: Saving service prometheus spec with placement smithi151=a;count:1 2024-03-16T14:31:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:15 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:15.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:15 smithi151 ceph-mon[30603]: pgmap v199: 132 pgs: 7 creating+peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.8 KiB/s wr, 138 op/s 2024-03-16T14:31:15.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[25218]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:15.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[25218]: Saving service prometheus spec with placement smithi151=a;count:1 2024-03-16T14:31:15.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:15.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[25218]: pgmap v199: 132 pgs: 7 creating+peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.8 KiB/s wr, 138 op/s 2024-03-16T14:31:15.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[38287]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:15.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[38287]: Saving service prometheus spec with placement smithi151=a;count:1 2024-03-16T14:31:15.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:15.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:15 smithi026 ceph-mon[38287]: pgmap v199: 132 pgs: 7 creating+peering, 125 active+clean; 581 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.8 KiB/s wr, 138 op/s 2024-03-16T14:31:15.922 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:15 smithi151 systemd[1]: Starting Ceph iscsi.iscsi.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:31:15.948 DEBUG:teuthology.orchestra.run.smithi151:prometheus.a> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@prometheus.a.service 2024-03-16T14:31:15.950 INFO:tasks.cephadm:Adding node-exporter.a on smithi026 2024-03-16T14:31:15.950 INFO:tasks.cephadm:Adding node-exporter.b on smithi151 2024-03-16T14:31:15.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply node-exporter '2;smithi026=a;smithi151=b' 2024-03-16T14:31:16.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 podman[67321]: 2024-03-16 14:31:16.047047233 +0000 UTC m=+0.028532804 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:31:16.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 podman[67321]: 2024-03-16 14:31:16.153985048 +0000 UTC m=+0.135470587 container create c51e5d771428c960a4ad7b4cff817955dc4ba0f8a1b317975a024eb65a43c24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-16T14:31:16.502 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:16.631 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 podman[67321]: 2024-03-16 14:31:16.389463656 +0000 UTC m=+0.370949198 container init c51e5d771428c960a4ad7b4cff817955dc4ba0f8a1b317975a024eb65a43c24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c) 2024-03-16T14:31:16.631 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 podman[67321]: 2024-03-16 14:31:16.392490787 +0000 UTC m=+0.373976327 container start c51e5d771428c960a4ad7b4cff817955dc4ba0f8a1b317975a024eb65a43c24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-16T14:31:16.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 bash[67321]: c51e5d771428c960a4ad7b4cff817955dc4ba0f8a1b317975a024eb65a43c24f 2024-03-16T14:31:16.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 systemd[1]: Started Ceph iscsi.iscsi.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:31:16.911 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug Started the configuration object watcher 2024-03-16T14:31:16.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug Checking for config object changes every 1s 2024-03-16T14:31:16.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:16 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug Processing osd blocklist entries for this node 2024-03-16T14:31:17.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug Reading the configuration object to update local LIO configuration 2024-03-16T14:31:17.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug Configuration does not have an entry for this host(smithi151.front.sepia.ceph.com) - nothing to define to LIO 2024-03-16T14:31:17.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 163 op/s 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:17.799 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:17 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/1339791017' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T14:31:18.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 163 op/s 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/1339791017' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 5.1 KiB/s wr, 163 op/s 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:18.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:17 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/1339791017' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-16T14:31:18.844 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled node-exporter update... 2024-03-16T14:31:19.587 DEBUG:teuthology.orchestra.run.smithi026:node-exporter.a> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.a.service 2024-03-16T14:31:19.590 DEBUG:teuthology.orchestra.run.smithi151:node-exporter.b> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.b.service 2024-03-16T14:31:19.592 INFO:tasks.cephadm:Adding alertmanager.a on smithi026 2024-03-16T14:31:19.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply alertmanager '1;smithi026=a' 2024-03-16T14:31:19.884 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:19.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:19 smithi151 ceph-mon[30603]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 141 op/s 2024-03-16T14:31:19.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:19 smithi151 ceph-mon[30603]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-16T14:31:19.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:19 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:19.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:19 smithi151 ceph-mon[30603]: osdmap e68: 8 total, 8 up, 8 in 2024-03-16T14:31:20.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[25218]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 141 op/s 2024-03-16T14:31:20.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[25218]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-16T14:31:20.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:20.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[25218]: osdmap e68: 8 total, 8 up, 8 in 2024-03-16T14:31:20.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[38287]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 141 op/s 2024-03-16T14:31:20.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[38287]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-16T14:31:20.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:20.006 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:19 smithi026 ceph-mon[38287]: osdmap e68: 8 total, 8 up, 8 in 2024-03-16T14:31:20.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:20 smithi151 ceph-mon[30603]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi026=a;smithi151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:20.968 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:20 smithi151 ceph-mon[30603]: Saving service node-exporter spec with placement smithi026=a;smithi151=b;count:2 2024-03-16T14:31:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:20 smithi026 ceph-mon[25218]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi026=a;smithi151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:20 smithi026 ceph-mon[25218]: Saving service node-exporter spec with placement smithi026=a;smithi151=b;count:2 2024-03-16T14:31:21.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:20 smithi026 ceph-mon[38287]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi026=a;smithi151=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:21.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:20 smithi026 ceph-mon[38287]: Saving service node-exporter spec with placement smithi026=a;smithi151=b;count:2 2024-03-16T14:31:21.739 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled alertmanager update... 2024-03-16T14:31:21.933 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:21 smithi151 ceph-mon[30603]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 126 op/s 2024-03-16T14:31:21.934 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:21 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:22.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:21 smithi026 ceph-mon[38287]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 126 op/s 2024-03-16T14:31:22.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:21 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:22.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:21 smithi026 ceph-mon[25218]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 126 op/s 2024-03-16T14:31:22.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:21 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:22.688 DEBUG:teuthology.orchestra.run.smithi026:alertmanager.a> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@alertmanager.a.service 2024-03-16T14:31:22.691 INFO:tasks.cephadm:Adding grafana.a on smithi151 2024-03-16T14:31:22.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph orch apply grafana '1;smithi151=a' 2024-03-16T14:31:22.944 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi026=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: Saving service alertmanager spec with placement smithi026=a;count:1 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: Deploying daemon prometheus.a on smithi151 2024-03-16T14:31:22.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:22 smithi151 ceph-mon[30603]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi026=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: Saving service alertmanager spec with placement smithi026=a;count:1 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: Deploying daemon prometheus.a on smithi151 2024-03-16T14:31:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[38287]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi026=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: Saving service alertmanager spec with placement smithi026=a;count:1 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: Deploying daemon prometheus.a on smithi151 2024-03-16T14:31:23.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:22 smithi026 ceph-mon[25218]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-16T14:31:24.807 INFO:teuthology.orchestra.run.smithi151.stdout:Scheduled grafana update... 2024-03-16T14:31:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:24 smithi151 ceph-mon[30603]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 921 B/s wr, 34 op/s 2024-03-16T14:31:25.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:24 smithi026 ceph-mon[38287]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 921 B/s wr, 34 op/s 2024-03-16T14:31:25.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:24 smithi026 ceph-mon[25218]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 921 B/s wr, 34 op/s 2024-03-16T14:31:26.125 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:25 smithi151 ceph-mon[30603]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:26.125 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:25 smithi151 ceph-mon[30603]: Saving service grafana spec with placement smithi151=a;count:1 2024-03-16T14:31:26.125 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:25 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:26.125 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:25 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:26.154 DEBUG:teuthology.orchestra.run.smithi151:grafana.a> sudo journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@grafana.a.service 2024-03-16T14:31:26.157 INFO:tasks.cephadm:Setting up client nodes... 2024-03-16T14:31:26.157 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T14:31:26.186 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[38287]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:26.186 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[38287]: Saving service grafana spec with placement smithi151=a;count:1 2024-03-16T14:31:26.186 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:26.186 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:26.186 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[25218]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi151=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:31:26.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[25218]: Saving service grafana spec with placement smithi151=a;count:1 2024-03-16T14:31:26.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:26.187 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:25 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:26.413 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:26.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:26 smithi151 ceph-mon[30603]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-03-16T14:31:26.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:26 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:31:27.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:26 smithi026 ceph-mon[38287]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-03-16T14:31:27.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:26 smithi026 ceph-mon[25218]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-03-16T14:31:28.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:27 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:28.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:27 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:28.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:27 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:28.365 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-03-16T14:31:28.365 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQBArfVlwWiKFRAAXxW7xvZOjTFhFQ2uMY/HNg== 2024-03-16T14:31:28.969 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:28 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/476739140' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:28.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:28 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/476739140' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T14:31:28.970 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:28 smithi026 ceph-mon[38287]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-16T14:31:28.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:28 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/476739140' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:28.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:28 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/476739140' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T14:31:28.970 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:28 smithi026 ceph-mon[25218]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-16T14:31:28.997 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:31:28.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-16T14:31:28.997 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-16T14:31:29.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-16T14:31:29.105 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:28 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/476739140' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:29.106 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:28 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/476739140' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T14:31:29.106 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:28 smithi151 ceph-mon[30603]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-16T14:31:30.095 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.b/config 2024-03-16T14:31:30.776 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-16T14:31:30.776 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-16T14:31:30.777 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:30 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-16T14:31:30.777 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:30 smithi151 ceph-mon[30603]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-16T14:31:31.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-16T14:31:31.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-16T14:31:31.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-16T14:31:31.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[38287]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-16T14:31:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-16T14:31:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-16T14:31:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-03-16T14:31:31.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:30 smithi026 ceph-mon[25218]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-16T14:31:31.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-16T14:31:31.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-16T14:31:31.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:31 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-16T14:31:31.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:31 smithi151 ceph-mon[30603]: osdmap e69: 8 total, 8 up, 8 in 2024-03-16T14:31:32.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-16T14:31:32.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-16T14:31:32.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-16T14:31:32.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[38287]: osdmap e69: 8 total, 8 up, 8 in 2024-03-16T14:31:32.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-16T14:31:32.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-16T14:31:32.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-03-16T14:31:32.006 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:31 smithi026 ceph-mon[25218]: osdmap e69: 8 total, 8 up, 8 in 2024-03-16T14:31:32.885 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:32 smithi151 ceph-mon[30603]: osdmap e70: 8 total, 8 up, 8 in 2024-03-16T14:31:32.885 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:32 smithi151 ceph-mon[30603]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-16T14:31:33.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:32 smithi026 ceph-mon[38287]: osdmap e70: 8 total, 8 up, 8 in 2024-03-16T14:31:33.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:32 smithi026 ceph-mon[38287]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-16T14:31:33.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:32 smithi026 ceph-mon[25218]: osdmap e70: 8 total, 8 up, 8 in 2024-03-16T14:31:33.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:32 smithi026 ceph-mon[25218]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-16T14:31:33.284 INFO:teuthology.orchestra.run.smithi151.stdout:[client.1] 2024-03-16T14:31:33.284 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQBFrfVlKlWoEBAAD63dDf0YnoW4hsc9lt5sVA== 2024-03-16T14:31:33.962 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:33 smithi151 ceph-mon[30603]: from='client.? 172.21.15.151:0/108686102' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:33.962 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:33 smithi151 ceph-mon[30603]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:33.962 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:33 smithi151 ceph-mon[30603]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T14:31:34.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:33 smithi026 ceph-mon[38287]: from='client.? 172.21.15.151:0/108686102' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:34.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:33 smithi026 ceph-mon[38287]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:34.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:33 smithi026 ceph-mon[38287]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T14:31:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:33 smithi026 ceph-mon[25218]: from='client.? 172.21.15.151:0/108686102' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:33 smithi026 ceph-mon[25218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-16T14:31:34.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:33 smithi026 ceph-mon[25218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-16T14:31:34.919 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:34 smithi151 ceph-mon[30603]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-03-16T14:31:34.952 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:31:34.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-16T14:31:34.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-16T14:31:35.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:34 smithi026 ceph-mon[38287]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-03-16T14:31:35.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:34 smithi026 ceph-mon[25218]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 1 objects/s recovering 2024-03-16T14:31:35.040 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-16T14:31:35.040 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-16T14:31:35.040 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-16T14:31:35.295 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:35.987 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:35 smithi151 systemd[1]: Starting Ceph prometheus.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:31:36.381 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 podman[68930]: 2024-03-16 14:31:36.016096938 +0000 UTC m=+0.018955775 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-16T14:31:36.381 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 podman[68930]: 2024-03-16 14:31:36.113989191 +0000 UTC m=+0.116848024 container create ab86899811bd9e43122c83c88e07e4e13d2721bb57a1885ce4d0188910254ae2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-16T14:31:36.381 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 podman[68930]: 2024-03-16 14:31:36.324344087 +0000 UTC m=+0.327202920 container init ab86899811bd9e43122c83c88e07e4e13d2721bb57a1885ce4d0188910254ae2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-16T14:31:36.381 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 podman[68930]: 2024-03-16 14:31:36.326998975 +0000 UTC m=+0.329857819 container start ab86899811bd9e43122c83c88e07e4e13d2721bb57a1885ce4d0188910254ae2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-16T14:31:36.381 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.351Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-16T14:31:36.381 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.351Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.352Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi151 (none))" 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.352Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.352Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.353Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.354Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.355Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.355Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.357Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.357Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.371µs 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.357Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.358Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.358Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.051µs wal_replay_duration=353.525µs wbl_replay_duration=143ns total_replay_duration=404.215µs 2024-03-16T14:31:36.382 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.358Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-16T14:31:36.383 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.358Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-16T14:31:36.383 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.358Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-16T14:31:36.383 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.378Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.027212ms db_storage=766ns remote_storage=997ns web_handler=446ns query_engine=604ns scrape=458.519µs scrape_sd=87.971µs notify=691ns notify_sd=1.344µs rules=19.224108ms tracing=5.99µs 2024-03-16T14:31:36.383 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.378Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-16T14:31:36.383 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:31:36.378Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-16T14:31:36.704 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 bash[68930]: ab86899811bd9e43122c83c88e07e4e13d2721bb57a1885ce4d0188910254ae2 2024-03-16T14:31:36.704 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:31:36 smithi151 systemd[1]: Started Ceph prometheus.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:31:36.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:31:37.249 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:31:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-16T14:31:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[25218]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 2 objects/s recovering 2024-03-16T14:31:37.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3632970105' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T14:31:37.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-16T14:31:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[38287]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 2 objects/s recovering 2024-03-16T14:31:37.506 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3632970105' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T14:31:37.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: ignoring --setuser ceph since I am not root 2024-03-16T14:31:37.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: ignoring --setgroup ceph since I am not root 2024-03-16T14:31:37.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-mgr[32455]: -- 172.21.15.151:0/86953038 <== mon.1 v2:172.21.15.151:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5583fe4d85a0 con 0x5583fe4acc00 2024-03-16T14:31:37.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' 2024-03-16T14:31:37.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-16T14:31:37.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-mon[30603]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 2 objects/s recovering 2024-03-16T14:31:37.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:37 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3632970105' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-16T14:31:37.933 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ignoring --setuser ceph since I am not root 2024-03-16T14:31:37.934 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:37 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ignoring --setgroup ceph since I am not root 2024-03-16T14:31:37.961 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6800","nonce":2389178901},{"type":"v1","addr":"172.21.15.26:6801","nonce":2389178901}]},"active_addr":"172.21.15.26:6801/2389178901","active_change":"2024-03-16T14:26:30.528966+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.26:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":494461760}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":1379875860}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":2130638101}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":3257022339}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.26:0","nonce":2077826243}]}]} 2024-03-16T14:31:37.968 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-16T14:31:37.968 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-16T14:31:37.968 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T14:31:38.216 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:38.222 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:38.114+0000 7f721c038200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T14:31:38.222 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:38.219+0000 7f721c038200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T14:31:38.486 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:38 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:38.092+0000 7f96ec0d7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-16T14:31:38.487 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:38 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:38.185+0000 7f96ec0d7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-16T14:31:38.824 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:38 smithi151 ceph-mon[30603]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-16T14:31:38.824 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:38 smithi151 ceph-mon[30603]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-16T14:31:38.900 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-mon[38287]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-16T14:31:38.900 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-mon[38287]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-16T14:31:38.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-mon[25218]: from='mgr.14150 172.21.15.26:0/4247387797' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-16T14:31:38.900 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-mon[25218]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-16T14:31:39.154 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:38 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:38.821+0000 7f96ec0d7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T14:31:39.154 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:38 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:38.914+0000 7f96ec0d7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T14:31:39.154 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.056+0000 7f96ec0d7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T14:31:39.155 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.151+0000 7f96ec0d7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T14:31:39.245 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:38.897+0000 7f721c038200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-16T14:31:39.245 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:38 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:38.995+0000 7f721c038200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-16T14:31:39.245 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:39 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:39.144+0000 7f721c038200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-16T14:31:39.486 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.233+0000 7f96ec0d7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T14:31:39.505 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:39 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:39.242+0000 7f721c038200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-16T14:31:39.505 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:39 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:39.328+0000 7f721c038200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-16T14:31:39.794 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.499+0000 7f96ec0d7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T14:31:39.794 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.609+0000 7f96ec0d7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T14:31:39.794 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.695+0000 7f96ec0d7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T14:31:39.794 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.791+0000 7f96ec0d7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T14:31:39.815 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:39 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:39.606+0000 7f721c038200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-16T14:31:39.815 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:39 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:39.722+0000 7f721c038200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-16T14:31:39.815 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:39 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:39.813+0000 7f721c038200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-16T14:31:40.091 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:31:40.092 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:39 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:39.918+0000 7f721c038200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-16T14:31:40.092 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":70,"fsid":"efcc9110-e3a0-11ee-95c9-87774f69a715","created":"2024-03-16T14:25:30.002920+0000","modified":"2024-03-16T14:31:31.619013+0000","last_up_change":"2024-03-16T14:30:51.676817+0000","last_in_change":"2024-03-16T14:30:33.385351+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T14:28:44.874373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-16T14:31:02.996731+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-16T14:31:03.241436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-16T14:31:05.010007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-16T14:31:06.951174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-16T14:31:09.020198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e773bb90-a94a-471e-8d8b-51a7cbada97b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6803","nonce":2042975011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6805","nonce":2042975011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6809","nonce":2042975011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6807","nonce":2042975011}]},"public_addr":"172.21.15.26:6803/2042975011","cluster_addr":"172.21.15.26:6805/2042975011","heartbeat_back_addr":"172.21.15.26:6809/2042975011","heartbeat_front_addr":"172.21.15.26:6807/2042975011","state":["exists","up"]},{"osd":1,"uuid":"79b74b66-d263-4eb1-af4c-76fcef030e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6811","nonce":2687555532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6813","nonce":2687555532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6817","nonce":2687555532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6815","nonce":2687555532}]},"public_addr":"172.21.15.26:6811/2687555532","cluster_addr":"172.21.15.26:6813/2687555532","heartbeat_back_addr":"172.21.15.26:6817/2687555532","heartbeat_front_addr":"172.21.15.26:6815/2687555532","state":["exists","up"]},{"osd":2,"uuid":"7a33de3b-ab23-4c5f-aeef-5688c34717db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6819","nonce":4025043318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6821","nonce":4025043318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6825","nonce":4025043318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6823","nonce":4025043318}]},"public_addr":"172.21.15.26:6819/4025043318","cluster_addr":"172.21.15.26:6821/4025043318","heartbeat_back_addr":"172.21.15.26:6825/4025043318","heartbeat_front_addr":"172.21.15.26:6823/4025043318","state":["exists","up"]},{"osd":3,"uuid":"b12f1441-6a57-485e-8bbc-7ae868ed9d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6827","nonce":3222197319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6829","nonce":3222197319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6833","nonce":3222197319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6831","nonce":3222197319}]},"public_addr":"172.21.15.26:6827/3222197319","cluster_addr":"172.21.15.26:6829/3222197319","heartbeat_back_addr":"172.21.15.26:6833/3222197319","heartbeat_front_addr":"172.21.15.26:6831/3222197319","state":["exists","up"]},{"osd":4,"uuid":"9abf790f-4b0c-4f91-893b-356b795f985d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6801","nonce":1799636760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6803","nonce":1799636760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6807","nonce":1799636760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6805","nonce":1799636760}]},"public_addr":"172.21.15.151:6801/1799636760","cluster_addr":"172.21.15.151:6803/1799636760","heartbeat_back_addr":"172.21.15.151:6807/1799636760","heartbeat_front_addr":"172.21.15.151:6805/1799636760","state":["exists","up"]},{"osd":5,"uuid":"ffea97ec-5a5f-4dc9-a5ba-a7d304280675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6809","nonce":7047105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6811","nonce":7047105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6815","nonce":7047105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6813","nonce":7047105}]},"public_addr":"172.21.15.151:6809/7047105","cluster_addr":"172.21.15.151:6811/7047105","heartbeat_back_addr":"172.21.15.151:6815/7047105","heartbeat_front_addr":"172.21.15.151:6813/7047105","state":["exists","up"]},{"osd":6,"uuid":"640fca26-3089-4b9a-9efd-0a054f336457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6817","nonce":2840379676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6819","nonce":2840379676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6823","nonce":2840379676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6821","nonce":2840379676}]},"public_addr":"172.21.15.151:6817/2840379676","cluster_addr":"172.21.15.151:6819/2840379676","heartbeat_back_addr":"172.21.15.151:6823/2840379676","heartbeat_front_addr":"172.21.15.151:6821/2840379676","state":["exists","up"]},{"osd":7,"uuid":"80b545b9-c39f-4330-9c35-82e904945f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6825","nonce":2379156800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6827","nonce":2379156800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6831","nonce":2379156800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6829","nonce":2379156800}]},"public_addr":"172.21.15.151:6825/2379156800","cluster_addr":"172.21.15.151:6827/2379156800","heartbeat_back_addr":"172.21.15.151:6831/2379156800","heartbeat_front_addr":"172.21.15.151:6829/2379156800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:27:51.608819+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:28:16.714427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:28:41.938355+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:07.808874+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:31.573352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:57.599378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:30:23.859894+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:30:49.882000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/929495780":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/1583238033":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/3218505860":"2024-03-17T14:26:30.528857+0000","172.21.15.26:6801/1667601179":"2024-03-17T14:26:04.256747+0000","172.21.15.26:6800/1667601179":"2024-03-17T14:26:04.256747+0000","172.21.15.26:6800/929495780":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/1857687384":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/659101530":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/3734138219":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/3185583976":"2024-03-17T14:26:30.528857+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T14:31:40.142 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:39.969+0000 7f96ec0d7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T14:31:40.405 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:40 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1146038444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:31:40.406 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:40 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1146038444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:31:40.406 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:40 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:40.100+0000 7f721c038200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-16T14:31:40.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:40 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1146038444' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:31:40.783 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-16T14:31:40.783 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-16T14:31:41.029 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:41.237 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:40 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:40.940+0000 7f96ec0d7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T14:31:41.237 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:41.069+0000 7f96ec0d7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T14:31:41.384 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:41 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:41.124+0000 7f721c038200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-16T14:31:41.384 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:41 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:41.264+0000 7f721c038200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-16T14:31:41.511 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:41.412+0000 7f96ec0d7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T14:31:41.710 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:41 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:41.613+0000 7f721c038200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-16T14:31:41.916 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:41.507+0000 7f96ec0d7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T14:31:41.916 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:41.718+0000 7f96ec0d7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:31:42.005 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:41 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:41.707+0000 7f721c038200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-16T14:31:42.005 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:41 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:41.912+0000 7f721c038200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:31:42.213 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:41.912+0000 7f96ec0d7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T14:31:42.371 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:42 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:42.105+0000 7f721c038200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-16T14:31:42.483 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:42.210+0000 7f96ec0d7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:31:42.703 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:42 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:42.423+0000 7f721c038200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-16T14:31:42.703 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:42 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:42.700+0000 7f721c038200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T14:31:42.736 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:42.480+0000 7f96ec0d7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-16T14:31:42.736 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:42.719+0000 7f96ec0d7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T14:31:42.814 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:31:42.814 INFO:teuthology.orchestra.run.smithi026.stdout:{"epoch":70,"fsid":"efcc9110-e3a0-11ee-95c9-87774f69a715","created":"2024-03-16T14:25:30.002920+0000","modified":"2024-03-16T14:31:31.619013+0000","last_up_change":"2024-03-16T14:30:51.676817+0000","last_in_change":"2024-03-16T14:30:33.385351+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-16T14:28:44.874373+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-16T14:31:02.996731+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-16T14:31:03.241436+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-16T14:31:05.010007+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-16T14:31:06.951174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-16T14:31:09.020198+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e773bb90-a94a-471e-8d8b-51a7cbada97b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6802","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6803","nonce":2042975011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6804","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6805","nonce":2042975011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6808","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6809","nonce":2042975011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6806","nonce":2042975011},{"type":"v1","addr":"172.21.15.26:6807","nonce":2042975011}]},"public_addr":"172.21.15.26:6803/2042975011","cluster_addr":"172.21.15.26:6805/2042975011","heartbeat_back_addr":"172.21.15.26:6809/2042975011","heartbeat_front_addr":"172.21.15.26:6807/2042975011","state":["exists","up"]},{"osd":1,"uuid":"79b74b66-d263-4eb1-af4c-76fcef030e09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6810","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6811","nonce":2687555532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6812","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6813","nonce":2687555532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6816","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6817","nonce":2687555532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6814","nonce":2687555532},{"type":"v1","addr":"172.21.15.26:6815","nonce":2687555532}]},"public_addr":"172.21.15.26:6811/2687555532","cluster_addr":"172.21.15.26:6813/2687555532","heartbeat_back_addr":"172.21.15.26:6817/2687555532","heartbeat_front_addr":"172.21.15.26:6815/2687555532","state":["exists","up"]},{"osd":2,"uuid":"7a33de3b-ab23-4c5f-aeef-5688c34717db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6818","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6819","nonce":4025043318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6820","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6821","nonce":4025043318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6824","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6825","nonce":4025043318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6822","nonce":4025043318},{"type":"v1","addr":"172.21.15.26:6823","nonce":4025043318}]},"public_addr":"172.21.15.26:6819/4025043318","cluster_addr":"172.21.15.26:6821/4025043318","heartbeat_back_addr":"172.21.15.26:6825/4025043318","heartbeat_front_addr":"172.21.15.26:6823/4025043318","state":["exists","up"]},{"osd":3,"uuid":"b12f1441-6a57-485e-8bbc-7ae868ed9d43","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6826","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6827","nonce":3222197319}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6828","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6829","nonce":3222197319}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6832","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6833","nonce":3222197319}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.26:6830","nonce":3222197319},{"type":"v1","addr":"172.21.15.26:6831","nonce":3222197319}]},"public_addr":"172.21.15.26:6827/3222197319","cluster_addr":"172.21.15.26:6829/3222197319","heartbeat_back_addr":"172.21.15.26:6833/3222197319","heartbeat_front_addr":"172.21.15.26:6831/3222197319","state":["exists","up"]},{"osd":4,"uuid":"9abf790f-4b0c-4f91-893b-356b795f985d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6801","nonce":1799636760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6803","nonce":1799636760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6807","nonce":1799636760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1799636760},{"type":"v1","addr":"172.21.15.151:6805","nonce":1799636760}]},"public_addr":"172.21.15.151:6801/1799636760","cluster_addr":"172.21.15.151:6803/1799636760","heartbeat_back_addr":"172.21.15.151:6807/1799636760","heartbeat_front_addr":"172.21.15.151:6805/1799636760","state":["exists","up"]},{"osd":5,"uuid":"ffea97ec-5a5f-4dc9-a5ba-a7d304280675","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6809","nonce":7047105}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6811","nonce":7047105}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6815","nonce":7047105}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":7047105},{"type":"v1","addr":"172.21.15.151:6813","nonce":7047105}]},"public_addr":"172.21.15.151:6809/7047105","cluster_addr":"172.21.15.151:6811/7047105","heartbeat_back_addr":"172.21.15.151:6815/7047105","heartbeat_front_addr":"172.21.15.151:6813/7047105","state":["exists","up"]},{"osd":6,"uuid":"640fca26-3089-4b9a-9efd-0a054f336457","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6817","nonce":2840379676}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6819","nonce":2840379676}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6823","nonce":2840379676}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":2840379676},{"type":"v1","addr":"172.21.15.151:6821","nonce":2840379676}]},"public_addr":"172.21.15.151:6817/2840379676","cluster_addr":"172.21.15.151:6819/2840379676","heartbeat_back_addr":"172.21.15.151:6823/2840379676","heartbeat_front_addr":"172.21.15.151:6821/2840379676","state":["exists","up"]},{"osd":7,"uuid":"80b545b9-c39f-4330-9c35-82e904945f61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6825","nonce":2379156800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6827","nonce":2379156800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6831","nonce":2379156800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":2379156800},{"type":"v1","addr":"172.21.15.151:6829","nonce":2379156800}]},"public_addr":"172.21.15.151:6825/2379156800","cluster_addr":"172.21.15.151:6827/2379156800","heartbeat_back_addr":"172.21.15.151:6831/2379156800","heartbeat_front_addr":"172.21.15.151:6829/2379156800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:27:51.608819+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:28:16.714427+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:28:41.938355+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:07.808874+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:31.573352+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:29:57.599378+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:30:23.859894+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-16T14:30:49.882000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.26:6801/929495780":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/1583238033":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/3218505860":"2024-03-17T14:26:30.528857+0000","172.21.15.26:6801/1667601179":"2024-03-17T14:26:04.256747+0000","172.21.15.26:6800/1667601179":"2024-03-17T14:26:04.256747+0000","172.21.15.26:6800/929495780":"2024-03-17T14:26:30.528857+0000","172.21.15.26:0/1857687384":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/659101530":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/3734138219":"2024-03-17T14:26:04.256747+0000","172.21.15.26:0/3185583976":"2024-03-17T14:26:30.528857+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-16T14:31:42.957 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:42 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/2719853261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:31:42.958 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:42 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2719853261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:31:43.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:42 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/2719853261' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-16T14:31:43.254 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:42 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:42.955+0000 7f721c038200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-16T14:31:43.574 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-16T14:31:43.574 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-16T14:31:43.574 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-16T14:31:43.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-16T14:31:43.575 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-16T14:31:43.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-16T14:31:43.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-16T14:31:43.576 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-16T14:31:43.986 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:43.531+0000 7f96ec0d7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T14:31:43.995 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:43 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:43.989+0000 7f721c038200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-16T14:31:44.083 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:44.095 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:44.423 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:44.102+0000 7f96ec0d7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T14:31:44.423 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:44.201+0000 7f96ec0d7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T14:31:44.423 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:44.420+0000 7f96ec0d7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T14:31:44.497 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[38287]: Standby manager daemon x restarted 2024-03-16T14:31:44.497 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[25218]: Standby manager daemon x restarted 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-mon[30603]: Standby manager daemon x restarted 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-mon[30603]: Standby manager daemon x started 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-mon[30603]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: [16/Mar/2024:14:31:44] ENGINE Bus STARTING 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: CherryPy Checker: 2024-03-16T14:31:44.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: The Application mounted at '' has an empty config. 2024-03-16T14:31:44.738 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: 2024-03-16T14:31:44.738 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: [16/Mar/2024:14:31:44] ENGINE Serving on http://:::9283 2024-03-16T14:31:44.738 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:31:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x[32451]: [16/Mar/2024:14:31:44] ENGINE Bus STARTED 2024-03-16T14:31:44.755 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:44.651+0000 7f721c038200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-16T14:31:44.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[25218]: Standby manager daemon x started 2024-03-16T14:31:44.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-16T14:31:44.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T14:31:44.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-16T14:31:44.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[25218]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T14:31:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[38287]: Standby manager daemon x started 2024-03-16T14:31:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-16T14:31:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-16T14:31:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-16T14:31:44.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-mon[38287]: from='mgr.? 172.21.15.151:0/2806950409' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-16T14:31:45.025 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:44.760+0000 7f721c038200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-16T14:31:45.025 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:44 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:44.987+0000 7f721c038200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-16T14:31:45.484 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:45.486 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:45.501 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:31:45] ENGINE Bus STARTING 2024-03-16T14:31:45.501 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: CherryPy Checker: 2024-03-16T14:31:45.501 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: The Application mounted at '' has an empty config. 2024-03-16T14:31:45.502 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:31:45.502 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:31:45] ENGINE Serving on http://:::9283 2024-03-16T14:31:45.502 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:31:45] ENGINE Bus STARTED 2024-03-16T14:31:45.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-16T14:31:45.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: Active manager daemon y restarted 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: Activating manager daemon y 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: osdmap e71: 8 total, 8 up, 8 in 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: mgrmap e19: y(active, starting, since 0.00730686s), standbys: x 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:31:45.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: Manager daemon y is now available 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:31:45.757 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: Active manager daemon y restarted 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: Activating manager daemon y 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: osdmap e71: 8 total, 8 up, 8 in 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: mgrmap e19: y(active, starting, since 0.00730686s), standbys: x 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:31:45.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: Manager daemon y is now available 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:31:45.759 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:31:45.760 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:45 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:31:45.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-16T14:31:45.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: Active manager daemon y restarted 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: Activating manager daemon y 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: osdmap e71: 8 total, 8 up, 8 in 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: mgrmap e19: y(active, starting, since 0.00730686s), standbys: x 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-16T14:31:45.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: Manager daemon y is now available 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:31:45.989 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:45 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-16T14:31:46.310 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:46.311 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:46.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:31:47.377 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[25218]: [16/Mar/2024:14:31:45] ENGINE Bus STARTING 2024-03-16T14:31:47.377 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[25218]: [16/Mar/2024:14:31:45] ENGINE Serving on https://172.21.15.26:7150 2024-03-16T14:31:47.377 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[25218]: mgrmap e20: y(active, since 1.01031s), standbys: x 2024-03-16T14:31:47.377 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[25218]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:47.377 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[25218]: [16/Mar/2024:14:31:46] ENGINE Serving on http://172.21.15.26:8765 2024-03-16T14:31:47.377 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[25218]: [16/Mar/2024:14:31:46] ENGINE Bus STARTED 2024-03-16T14:31:47.378 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[38287]: [16/Mar/2024:14:31:45] ENGINE Bus STARTING 2024-03-16T14:31:47.378 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[38287]: [16/Mar/2024:14:31:45] ENGINE Serving on https://172.21.15.26:7150 2024-03-16T14:31:47.378 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[38287]: mgrmap e20: y(active, since 1.01031s), standbys: x 2024-03-16T14:31:47.378 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[38287]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:47.378 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[38287]: [16/Mar/2024:14:31:46] ENGINE Serving on http://172.21.15.26:8765 2024-03-16T14:31:47.378 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:47 smithi026 ceph-mon[38287]: [16/Mar/2024:14:31:46] ENGINE Bus STARTED 2024-03-16T14:31:47.429 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:47 smithi151 ceph-mon[30603]: [16/Mar/2024:14:31:45] ENGINE Bus STARTING 2024-03-16T14:31:47.429 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:47 smithi151 ceph-mon[30603]: [16/Mar/2024:14:31:45] ENGINE Serving on https://172.21.15.26:7150 2024-03-16T14:31:47.429 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:47 smithi151 ceph-mon[30603]: mgrmap e20: y(active, since 1.01031s), standbys: x 2024-03-16T14:31:47.429 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:47 smithi151 ceph-mon[30603]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:47.429 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:47 smithi151 ceph-mon[30603]: [16/Mar/2024:14:31:46] ENGINE Serving on http://172.21.15.26:8765 2024-03-16T14:31:47.429 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:47 smithi151 ceph-mon[30603]: [16/Mar/2024:14:31:46] ENGINE Bus STARTED 2024-03-16T14:31:48.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:48 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:48.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:48 smithi026 ceph-mon[25218]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-16T14:31:48.255 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:48 smithi026 ceph-mon[25218]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:48.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:48 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:48.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:48 smithi026 ceph-mon[38287]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-16T14:31:48.255 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:48 smithi026 ceph-mon[38287]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:48.351 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:48 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:48.351 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:48 smithi151 ceph-mon[30603]: mgrmap e21: y(active, since 2s), standbys: x 2024-03-16T14:31:48.351 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:48 smithi151 ceph-mon[30603]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:48.620 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:48 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:31:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:31:50.447 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[38287]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:50.447 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[38287]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-16T14:31:50.447 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:50.447 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:50.447 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[25218]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:50.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[25218]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-16T14:31:50.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:50.448 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:50 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:50.486 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:50 smithi151 ceph-mon[30603]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:50.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:50 smithi151 ceph-mon[30603]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-16T14:31:50.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:50 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:50.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:50 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:50.851 INFO:teuthology.orchestra.run.smithi026.stdout:197568495633 2024-03-16T14:31:50.852 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-16T14:31:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi151 to 3760M 2024-03-16T14:31:51.738 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:51 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi151 to 3760M 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:31:51.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi151 to 3760M 2024-03-16T14:31:51.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:51 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:31:52.677 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:52 smithi026 ceph-mon[38287]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:52.677 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:52 smithi026 ceph-mon[25218]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:52.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:52 smithi151 ceph-mon[30603]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-16T14:31:54.449 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:31:54.475 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:54 smithi026 ceph-mon[38287]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-16T14:31:54.475 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:54 smithi026 ceph-mon[25218]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-16T14:31:54.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:54 smithi151 ceph-mon[30603]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-16T14:31:55.486 INFO:teuthology.orchestra.run.smithi026.stdout:128849018910 2024-03-16T14:31:55.487 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-16T14:31:56.291 INFO:teuthology.orchestra.run.smithi026.stdout:163208757272 2024-03-16T14:31:56.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-16T14:31:56.515 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:56 smithi026 ceph-mon[38287]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-16T14:31:56.515 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:56 smithi026 ceph-mon[25218]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-16T14:31:56.727 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:56 smithi151 ceph-mon[30603]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-16T14:31:56.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:31:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:31:58.614 INFO:teuthology.orchestra.run.smithi026.stdout:77309411369 2024-03-16T14:31:58.615 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-16T14:31:58.722 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:58 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:58.723 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:31:58 smithi026 ceph-mon[38287]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-16T14:31:58.723 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:31:58 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:31:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:31:58.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:58 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:58.723 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:31:58 smithi026 ceph-mon[25218]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-16T14:31:58.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:58 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:31:58.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:31:58 smithi151 ceph-mon[30603]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-16T14:31:59.314 INFO:teuthology.orchestra.run.smithi026.stdout:107374182434 2024-03-16T14:31:59.315 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-16T14:31:59.820 INFO:teuthology.orchestra.run.smithi026.stdout:231928233998 2024-03-16T14:31:59.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-16T14:32:00.419 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:00.419 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:00.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:00 smithi026 ceph-mon[38287]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:00.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:00 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:00.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:00 smithi026 ceph-mon[25218]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:00.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:00 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:00.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:00 smithi151 ceph-mon[30603]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:00.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:00 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:01.806 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:02.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:02 smithi026 ceph-mon[38287]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:02.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:02 smithi026 ceph-mon[25218]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:02.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:02 smithi151 ceph-mon[30603]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:02.887 INFO:teuthology.orchestra.run.smithi026.stdout:55834574894 2024-03-16T14:32:02.887 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-16T14:32:03.593 INFO:teuthology.orchestra.run.smithi026.stdout:34359738420 2024-03-16T14:32:03.593 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-16T14:32:04.163 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:04.175 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:04.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:04 smithi026 ceph-mon[38287]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:04.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:04 smithi026 ceph-mon[25218]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:04.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:04 smithi151 ceph-mon[30603]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-16T14:32:06.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:06 smithi151 ceph-mon[30603]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:06.754 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:06 smithi026 ceph-mon[25218]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:06.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:06 smithi026 ceph-mon[38287]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:06.757 INFO:teuthology.orchestra.run.smithi026.stdout:77309411370 2024-03-16T14:32:07.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:32:06 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:32:07.734 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:07 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3455978140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T14:32:07.735 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:07 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3455978140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T14:32:07.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:07 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3455978140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-16T14:32:07.835 INFO:teuthology.orchestra.run.smithi026.stdout:128849018913 2024-03-16T14:32:08.505 INFO:teuthology.orchestra.run.smithi026.stdout:197568495637 2024-03-16T14:32:08.518 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:08 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:08.518 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:08 smithi026 ceph-mon[25218]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:08.518 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:08 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3271474650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T14:32:08.518 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:08 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:08.518 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:08 smithi026 ceph-mon[38287]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:08.518 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:08 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3271474650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T14:32:08.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:08 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:08.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:08 smithi151 ceph-mon[30603]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:08.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:08 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3271474650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-16T14:32:08.892 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:32:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:32:08.921 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018913 for osd.4 2024-03-16T14:32:08.922 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:09.486 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411370 for osd.2 2024-03-16T14:32:09.486 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:09.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:09 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/2795136564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T14:32:09.748 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:09 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2795136564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T14:32:09.748 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:09 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/2795136564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-16T14:32:10.607 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:10 smithi026 ceph-mon[38287]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:10.607 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:10 smithi026 ceph-mon[25218]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:10.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:10 smithi151 ceph-mon[30603]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:10.855 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495633 got 197568495637 for osd.6 2024-03-16T14:32:10.855 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:11.543 INFO:teuthology.orchestra.run.smithi026.stdout:163208757276 2024-03-16T14:32:11.998 INFO:teuthology.orchestra.run.smithi026.stdout:231928234001 2024-03-16T14:32:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:12.219 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:12.688 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:12 smithi026 ceph-mon[38287]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-16T14:32:12.689 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:12 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/1453020649' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T14:32:12.689 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:12 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/3380297170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T14:32:12.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:12 smithi026 ceph-mon[25218]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-16T14:32:12.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:12 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/1453020649' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T14:32:12.689 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:12 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/3380297170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T14:32:12.721 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234001 for osd.7 2024-03-16T14:32:12.721 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:12.736 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:12 smithi151 ceph-mon[30603]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-16T14:32:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:12 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/1453020649' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-16T14:32:12.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:12 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/3380297170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-16T14:32:13.194 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:13.224 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757276 for osd.5 2024-03-16T14:32:13.224 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:14.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:14 smithi026 ceph-mon[38287]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:14.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:14 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:14.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:14 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:14.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:14 smithi026 ceph-mon[25218]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:14.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:14 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:14.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:14 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:14.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:14 smithi151 ceph-mon[30603]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:14.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:14 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:14.737 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:14 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:15.583 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:15 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:15.584 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:15 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:15.914 INFO:teuthology.orchestra.run.smithi026.stdout:34359738422 2024-03-16T14:32:15.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:15 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:16.748 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:16.748 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/289804426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T14:32:16.748 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:32:16.749 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/289804426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/289804426' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:32:16.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:16 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:32:17.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:32:16 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: Adjusting osd_memory_target on smithi026 to 3761M 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.833 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: Adjusting osd_memory_target on smithi026 to 3761M 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.834 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:17 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: Adjusting osd_memory_target on smithi026 to 3761M 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: Updating smithi026:/etc/ceph/ceph.conf 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: Updating smithi151:/etc/ceph/ceph.conf 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:17.988 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:17 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:18.071 INFO:teuthology.orchestra.run.smithi026.stdout:107374182440 2024-03-16T14:32:18.404 INFO:teuthology.orchestra.run.smithi026.stdout:55834574898 2024-03-16T14:32:18.755 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:32:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:32:18.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:32:18.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:32:18.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:18.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:32:18.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:32:18.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: Deploying daemon alertmanager.a on smithi026 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/4069077802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/2292626033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: Deploying daemon alertmanager.a on smithi026 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/4069077802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T14:32:18.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:18 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2292626033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T14:32:18.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: Updating smithi026:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:32:19.070 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-03-16T14:32:19.070 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:19.070 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: Updating smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:32:19.071 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: Updating smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-16T14:32:19.071 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:19.071 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: Deploying daemon alertmanager.a on smithi026 2024-03-16T14:32:19.071 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/4069077802' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-16T14:32:19.071 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:18 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/2292626033' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-16T14:32:19.446 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738422 for osd.0 2024-03-16T14:32:19.447 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:20.074 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574898 for osd.1 2024-03-16T14:32:20.074 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:20.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182440 for osd.3 2024-03-16T14:32:20.657 DEBUG:teuthology.parallel:result is None 2024-03-16T14:32:20.657 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-16T14:32:20.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T14:32:20.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:20 smithi151 ceph-mon[30603]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:21.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:20 smithi026 ceph-mon[25218]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:21.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:20 smithi026 ceph-mon[38287]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:21.358 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:22.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:22 smithi151 ceph-mon[30603]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:23.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:22 smithi026 ceph-mon[25218]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:23.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:22 smithi026 ceph-mon[38287]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:24.654 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:24 smithi026 ceph-mon[25218]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:24.654 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:24 smithi026 ceph-mon[38287]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:24.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:24 smithi151 ceph-mon[30603]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:26.841 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:26 smithi026 ceph-mon[38287]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:26.842 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:26 smithi026 ceph-mon[25218]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:26.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:26 smithi151 ceph-mon[30603]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:26.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:32:26 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:32:27.106 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:32:27.107 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 podman[81663]: 2024-03-16 14:32:27.075190834 +0000 UTC m=+0.123257559 volume create 82afb6ffc688e9addaddb43a48bfcb38633f6adec779e4aac5687c896468f545 2024-03-16T14:32:27.112 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-16T14:32:27.392 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 podman[81663]: 2024-03-16 14:32:27.192546639 +0000 UTC m=+0.240613365 container create 8a70f394d4fab3b617e0acc11a25f4f10452de7739385693edfe6a3649f79a19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 podman[81663]: 2024-03-16 14:32:27.495095141 +0000 UTC m=+0.543161883 container init 8a70f394d4fab3b617e0acc11a25f4f10452de7739385693edfe6a3649f79a19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 podman[81663]: 2024-03-16 14:32:27.497664192 +0000 UTC m=+0.545730927 container start 8a70f394d4fab3b617e0acc11a25f4f10452de7739385693edfe6a3649f79a19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.510Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.510Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.511Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.26 port=9094 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.512Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.536Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.536Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-16T14:32:27.650 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.539Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-16T14:32:27.651 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:27.539Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-16T14:32:27.651 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 bash[81663]: 8a70f394d4fab3b617e0acc11a25f4f10452de7739385693edfe6a3649f79a19 2024-03-16T14:32:27.651 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:27 smithi026 systemd[1]: Started Ceph alertmanager.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:32:27.915 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:32:27] ENGINE Bus STOPPING 2024-03-16T14:32:27.915 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:32:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-16T14:32:27.916 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:32:27] ENGINE Bus STOPPED 2024-03-16T14:32:27.916 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:32:27] ENGINE Bus STARTING 2024-03-16T14:32:27.916 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:32:27] ENGINE Serving on http://:::9283 2024-03-16T14:32:27.916 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:27 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:32:27] ENGINE Bus STARTED 2024-03-16T14:32:28.457 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-03-16T14:32:27.029582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1039,"num_read_kb":882,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":660,"ondisk_log_size":660,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":236628,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749691820,"statfs":{"total":767926730752,"available":767684423680,"internally_reserved":0,"allocated":8187904,"data_stored":4434104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002983"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008286+0000","last_change":"2024-03-16T14:31:10.719085+0000","last_active":"2024-03-16T14:31:45.008286+0000","last_peered":"2024-03-16T14:31:45.008286+0000","last_clean":"2024-03-16T14:31:45.008286+0000","last_became_active":"2024-03-16T14:31:10.718694+0000","last_became_peered":"2024-03-16T14:31:10.718694+0000","last_unstale":"2024-03-16T14:31:45.008286+0000","last_undegraded":"2024-03-16T14:31:45.008286+0000","last_fullsized":"2024-03-16T14:31:45.008286+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:47.412379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269048+0000","last_change":"2024-03-16T14:31:08.715961+0000","last_active":"2024-03-16T14:31:45.269048+0000","last_peered":"2024-03-16T14:31:45.269048+0000","last_clean":"2024-03-16T14:31:45.269048+0000","last_became_active":"2024-03-16T14:31:08.715607+0000","last_became_peered":"2024-03-16T14:31:08.715607+0000","last_unstale":"2024-03-16T14:31:45.269048+0000","last_undegraded":"2024-03-16T14:31:45.269048+0000","last_fullsized":"2024-03-16T14:31:45.269048+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:11.485717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010516+0000","last_change":"2024-03-16T14:31:04.671856+0000","last_active":"2024-03-16T14:31:45.010516+0000","last_peered":"2024-03-16T14:31:45.010516+0000","last_clean":"2024-03-16T14:31:45.010516+0000","last_became_active":"2024-03-16T14:31:04.671247+0000","last_became_peered":"2024-03-16T14:31:04.671247+0000","last_unstale":"2024-03-16T14:31:45.010516+0000","last_undegraded":"2024-03-16T14:31:45.010516+0000","last_fullsized":"2024-03-16T14:31:45.010516+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:07:17.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.007949+0000","last_change":"2024-03-16T14:31:06.697380+0000","last_active":"2024-03-16T14:31:45.007949+0000","last_peered":"2024-03-16T14:31:45.007949+0000","last_clean":"2024-03-16T14:31:45.007949+0000","last_became_active":"2024-03-16T14:31:06.697044+0000","last_became_peered":"2024-03-16T14:31:06.697044+0000","last_unstale":"2024-03-16T14:31:45.007949+0000","last_undegraded":"2024-03-16T14:31:45.007949+0000","last_fullsized":"2024-03-16T14:31:45.007949+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:37:22.253099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270099+0000","last_change":"2024-03-16T14:31:11.206078+0000","last_active":"2024-03-16T14:31:45.270099+0000","last_peered":"2024-03-16T14:31:45.270099+0000","last_clean":"2024-03-16T14:31:45.270099+0000","last_became_active":"2024-03-16T14:31:11.205524+0000","last_became_peered":"2024-03-16T14:31:11.205524+0000","last_unstale":"2024-03-16T14:31:45.270099+0000","last_undegraded":"2024-03-16T14:31:45.270099+0000","last_fullsized":"2024-03-16T14:31:45.270099+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:24:57.860915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.007952+0000","last_change":"2024-03-16T14:31:08.706796+0000","last_active":"2024-03-16T14:31:45.007952+0000","last_peered":"2024-03-16T14:31:45.007952+0000","last_clean":"2024-03-16T14:31:45.007952+0000","last_became_active":"2024-03-16T14:31:08.706445+0000","last_became_peered":"2024-03-16T14:31:08.706445+0000","last_unstale":"2024-03-16T14:31:45.007952+0000","last_undegraded":"2024-03-16T14:31:45.007952+0000","last_fullsized":"2024-03-16T14:31:45.007952+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:48.515244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270635+0000","last_change":"2024-03-16T14:31:04.669846+0000","last_active":"2024-03-16T14:31:45.270635+0000","last_peered":"2024-03-16T14:31:45.270635+0000","last_clean":"2024-03-16T14:31:45.270635+0000","last_became_active":"2024-03-16T14:31:04.669538+0000","last_became_peered":"2024-03-16T14:31:04.669538+0000","last_unstale":"2024-03-16T14:31:45.270635+0000","last_undegraded":"2024-03-16T14:31:45.270635+0000","last_fullsized":"2024-03-16T14:31:45.270635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:21.155953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269692+0000","last_change":"2024-03-16T14:31:06.701679+0000","last_active":"2024-03-16T14:31:45.269692+0000","last_peered":"2024-03-16T14:31:45.269692+0000","last_clean":"2024-03-16T14:31:45.269692+0000","last_became_active":"2024-03-16T14:31:06.701250+0000","last_became_peered":"2024-03-16T14:31:06.701250+0000","last_unstale":"2024-03-16T14:31:45.269692+0000","last_undegraded":"2024-03-16T14:31:45.269692+0000","last_fullsized":"2024-03-16T14:31:45.269692+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:58:50.977684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008545+0000","last_change":"2024-03-16T14:31:10.720081+0000","last_active":"2024-03-16T14:31:45.008545+0000","last_peered":"2024-03-16T14:31:45.008545+0000","last_clean":"2024-03-16T14:31:45.008545+0000","last_became_active":"2024-03-16T14:31:10.719496+0000","last_became_peered":"2024-03-16T14:31:10.719496+0000","last_unstale":"2024-03-16T14:31:45.008545+0000","last_undegraded":"2024-03-16T14:31:45.008545+0000","last_fullsized":"2024-03-16T14:31:45.008545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:39.356092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113846+0000","last_change":"2024-03-16T14:31:08.708031+0000","last_active":"2024-03-16T14:31:45.113846+0000","last_peered":"2024-03-16T14:31:45.113846+0000","last_clean":"2024-03-16T14:31:45.113846+0000","last_became_active":"2024-03-16T14:31:08.707031+0000","last_became_peered":"2024-03-16T14:31:08.707031+0000","last_unstale":"2024-03-16T14:31:45.113846+0000","last_undegraded":"2024-03-16T14:31:45.113846+0000","last_fullsized":"2024-03-16T14:31:45.113846+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:49:11.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114025+0000","last_change":"2024-03-16T14:31:04.670431+0000","last_active":"2024-03-16T14:31:45.114025+0000","last_peered":"2024-03-16T14:31:45.114025+0000","last_clean":"2024-03-16T14:31:45.114025+0000","last_became_active":"2024-03-16T14:31:04.670134+0000","last_became_peered":"2024-03-16T14:31:04.670134+0000","last_unstale":"2024-03-16T14:31:45.114025+0000","last_undegraded":"2024-03-16T14:31:45.114025+0000","last_fullsized":"2024-03-16T14:31:45.114025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:14:22.548560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008542+0000","last_change":"2024-03-16T14:31:06.703942+0000","last_active":"2024-03-16T14:31:45.008542+0000","last_peered":"2024-03-16T14:31:45.008542+0000","last_clean":"2024-03-16T14:31:45.008542+0000","last_became_active":"2024-03-16T14:31:06.703579+0000","last_became_peered":"2024-03-16T14:31:06.703579+0000","last_unstale":"2024-03-16T14:31:45.008542+0000","last_undegraded":"2024-03-16T14:31:45.008542+0000","last_fullsized":"2024-03-16T14:31:45.008542+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:58:51.973773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113685+0000","last_change":"2024-03-16T14:31:10.726539+0000","last_active":"2024-03-16T14:31:45.113685+0000","last_peered":"2024-03-16T14:31:45.113685+0000","last_clean":"2024-03-16T14:31:45.113685+0000","last_became_active":"2024-03-16T14:31:10.726241+0000","last_became_peered":"2024-03-16T14:31:10.726241+0000","last_unstale":"2024-03-16T14:31:45.113685+0000","last_undegraded":"2024-03-16T14:31:45.113685+0000","last_fullsized":"2024-03-16T14:31:45.113685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:12.298340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128686+0000","last_change":"2024-03-16T14:31:08.708166+0000","last_active":"2024-03-16T14:31:45.128686+0000","last_peered":"2024-03-16T14:31:45.128686+0000","last_clean":"2024-03-16T14:31:45.128686+0000","last_became_active":"2024-03-16T14:31:08.707798+0000","last_became_peered":"2024-03-16T14:31:08.707798+0000","last_unstale":"2024-03-16T14:31:45.128686+0000","last_undegraded":"2024-03-16T14:31:45.128686+0000","last_fullsized":"2024-03-16T14:31:45.128686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:03:51.224667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114019+0000","last_change":"2024-03-16T14:31:04.674726+0000","last_active":"2024-03-16T14:31:45.114019+0000","last_peered":"2024-03-16T14:31:45.114019+0000","last_clean":"2024-03-16T14:31:45.114019+0000","last_became_active":"2024-03-16T14:31:04.674469+0000","last_became_peered":"2024-03-16T14:31:04.674469+0000","last_unstale":"2024-03-16T14:31:45.114019+0000","last_undegraded":"2024-03-16T14:31:45.114019+0000","last_fullsized":"2024-03-16T14:31:45.114019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:15.554153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008949+0000","last_change":"2024-03-16T14:31:06.703886+0000","last_active":"2024-03-16T14:31:45.008949+0000","last_peered":"2024-03-16T14:31:45.008949+0000","last_clean":"2024-03-16T14:31:45.008949+0000","last_became_active":"2024-03-16T14:31:06.703513+0000","last_became_peered":"2024-03-16T14:31:06.703513+0000","last_unstale":"2024-03-16T14:31:45.008949+0000","last_undegraded":"2024-03-16T14:31:45.008949+0000","last_fullsized":"2024-03-16T14:31:45.008949+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:54.108322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010254+0000","last_change":"2024-03-16T14:31:11.202040+0000","last_active":"2024-03-16T14:31:45.010254+0000","last_peered":"2024-03-16T14:31:45.010254+0000","last_clean":"2024-03-16T14:31:45.010254+0000","last_became_active":"2024-03-16T14:31:11.201729+0000","last_became_peered":"2024-03-16T14:31:11.201729+0000","last_unstale":"2024-03-16T14:31:45.010254+0000","last_undegraded":"2024-03-16T14:31:45.010254+0000","last_fullsized":"2024-03-16T14:31:45.010254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:22:44.615838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270280+0000","last_change":"2024-03-16T14:31:08.716241+0000","last_active":"2024-03-16T14:31:45.270280+0000","last_peered":"2024-03-16T14:31:45.270280+0000","last_clean":"2024-03-16T14:31:45.270280+0000","last_became_active":"2024-03-16T14:31:08.715713+0000","last_became_peered":"2024-03-16T14:31:08.715713+0000","last_unstale":"2024-03-16T14:31:45.270280+0000","last_undegraded":"2024-03-16T14:31:45.270280+0000","last_fullsized":"2024-03-16T14:31:45.270280+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:18.500434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011099+0000","last_change":"2024-03-16T14:31:04.677477+0000","last_active":"2024-03-16T14:31:45.011099+0000","last_peered":"2024-03-16T14:31:45.011099+0000","last_clean":"2024-03-16T14:31:45.011099+0000","last_became_active":"2024-03-16T14:31:04.677169+0000","last_became_peered":"2024-03-16T14:31:04.677169+0000","last_unstale":"2024-03-16T14:31:45.011099+0000","last_undegraded":"2024-03-16T14:31:45.011099+0000","last_fullsized":"2024-03-16T14:31:45.011099+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:14.367957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121930+0000","last_change":"2024-03-16T14:31:06.694617+0000","last_active":"2024-03-16T14:31:45.121930+0000","last_peered":"2024-03-16T14:31:45.121930+0000","last_clean":"2024-03-16T14:31:45.121930+0000","last_became_active":"2024-03-16T14:31:06.694260+0000","last_became_peered":"2024-03-16T14:31:06.694260+0000","last_unstale":"2024-03-16T14:31:45.121930+0000","last_undegraded":"2024-03-16T14:31:45.121930+0000","last_fullsized":"2024-03-16T14:31:45.121930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:19.724889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270937+0000","last_change":"2024-03-16T14:31:11.202242+0000","last_active":"2024-03-16T14:31:45.270937+0000","last_peered":"2024-03-16T14:31:45.270937+0000","last_clean":"2024-03-16T14:31:45.270937+0000","last_became_active":"2024-03-16T14:31:11.201849+0000","last_became_peered":"2024-03-16T14:31:11.201849+0000","last_unstale":"2024-03-16T14:31:45.270937+0000","last_undegraded":"2024-03-16T14:31:45.270937+0000","last_fullsized":"2024-03-16T14:31:45.270937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:33:04.048437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009906+0000","last_change":"2024-03-16T14:31:08.708199+0000","last_active":"2024-03-16T14:31:45.009906+0000","last_peered":"2024-03-16T14:31:45.009906+0000","last_clean":"2024-03-16T14:31:45.009906+0000","last_became_active":"2024-03-16T14:31:08.707881+0000","last_became_peered":"2024-03-16T14:31:08.707881+0000","last_unstale":"2024-03-16T14:31:45.009906+0000","last_undegraded":"2024-03-16T14:31:45.009906+0000","last_fullsized":"2024-03-16T14:31:45.009906+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:42.193528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008670+0000","last_change":"2024-03-16T14:31:04.674838+0000","last_active":"2024-03-16T14:31:45.008670+0000","last_peered":"2024-03-16T14:31:45.008670+0000","last_clean":"2024-03-16T14:31:45.008670+0000","last_became_active":"2024-03-16T14:31:04.674538+0000","last_became_peered":"2024-03-16T14:31:04.674538+0000","last_unstale":"2024-03-16T14:31:45.008670+0000","last_undegraded":"2024-03-16T14:31:45.008670+0000","last_fullsized":"2024-03-16T14:31:45.008670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:09.972153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":70,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270208+0000","last_change":"2024-03-16T14:31:31.638092+0000","last_active":"2024-03-16T14:31:45.270208+0000","last_peered":"2024-03-16T14:31:45.270208+0000","last_clean":"2024-03-16T14:31:45.270208+0000","last_became_active":"2024-03-16T14:31:31.631485+0000","last_became_peered":"2024-03-16T14:31:31.631485+0000","last_unstale":"2024-03-16T14:31:45.270208+0000","last_undegraded":"2024-03-16T14:31:45.270208+0000","last_fullsized":"2024-03-16T14:31:45.270208+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:46.750504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128904+0000","last_change":"2024-03-16T14:31:10.721874+0000","last_active":"2024-03-16T14:31:45.128904+0000","last_peered":"2024-03-16T14:31:45.128904+0000","last_clean":"2024-03-16T14:31:45.128904+0000","last_became_active":"2024-03-16T14:31:10.721564+0000","last_became_peered":"2024-03-16T14:31:10.721564+0000","last_unstale":"2024-03-16T14:31:45.128904+0000","last_undegraded":"2024-03-16T14:31:45.128904+0000","last_fullsized":"2024-03-16T14:31:45.128904+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:36.459882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734558+0000","last_change":"2024-03-16T14:31:08.714781+0000","last_active":"2024-03-16T14:32:19.734558+0000","last_peered":"2024-03-16T14:32:19.734558+0000","last_clean":"2024-03-16T14:32:19.734558+0000","last_became_active":"2024-03-16T14:31:08.712843+0000","last_became_peered":"2024-03-16T14:31:08.712843+0000","last_unstale":"2024-03-16T14:32:19.734558+0000","last_undegraded":"2024-03-16T14:32:19.734558+0000","last_fullsized":"2024-03-16T14:32:19.734558+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:46:04.108601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269150+0000","last_change":"2024-03-16T14:31:04.678511+0000","last_active":"2024-03-16T14:31:45.269150+0000","last_peered":"2024-03-16T14:31:45.269150+0000","last_clean":"2024-03-16T14:31:45.269150+0000","last_became_active":"2024-03-16T14:31:04.678092+0000","last_became_peered":"2024-03-16T14:31:04.678092+0000","last_unstale":"2024-03-16T14:31:45.269150+0000","last_undegraded":"2024-03-16T14:31:45.269150+0000","last_fullsized":"2024-03-16T14:31:45.269150+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:47:12.439794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011259+0000","last_change":"2024-03-16T14:31:06.693954+0000","last_active":"2024-03-16T14:31:45.011259+0000","last_peered":"2024-03-16T14:31:45.011259+0000","last_clean":"2024-03-16T14:31:45.011259+0000","last_became_active":"2024-03-16T14:31:06.693583+0000","last_became_peered":"2024-03-16T14:31:06.693583+0000","last_unstale":"2024-03-16T14:31:45.011259+0000","last_undegraded":"2024-03-16T14:31:45.011259+0000","last_fullsized":"2024-03-16T14:31:45.011259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:02:07.726396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270379+0000","last_change":"2024-03-16T14:31:10.720149+0000","last_active":"2024-03-16T14:31:45.270379+0000","last_peered":"2024-03-16T14:31:45.270379+0000","last_clean":"2024-03-16T14:31:45.270379+0000","last_became_active":"2024-03-16T14:31:10.719813+0000","last_became_peered":"2024-03-16T14:31:10.719813+0000","last_unstale":"2024-03-16T14:31:45.270379+0000","last_undegraded":"2024-03-16T14:31:45.270379+0000","last_fullsized":"2024-03-16T14:31:45.270379+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:29:22.913842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011559+0000","last_change":"2024-03-16T14:31:08.709707+0000","last_active":"2024-03-16T14:31:45.011559+0000","last_peered":"2024-03-16T14:31:45.011559+0000","last_clean":"2024-03-16T14:31:45.011559+0000","last_became_active":"2024-03-16T14:31:08.709325+0000","last_became_peered":"2024-03-16T14:31:08.709325+0000","last_unstale":"2024-03-16T14:31:45.011559+0000","last_undegraded":"2024-03-16T14:31:45.011559+0000","last_fullsized":"2024-03-16T14:31:45.011559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:17.351327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268725+0000","last_change":"2024-03-16T14:31:04.677294+0000","last_active":"2024-03-16T14:31:45.268725+0000","last_peered":"2024-03-16T14:31:45.268725+0000","last_clean":"2024-03-16T14:31:45.268725+0000","last_became_active":"2024-03-16T14:31:04.676996+0000","last_became_peered":"2024-03-16T14:31:04.676996+0000","last_unstale":"2024-03-16T14:31:45.268725+0000","last_undegraded":"2024-03-16T14:31:45.268725+0000","last_fullsized":"2024-03-16T14:31:45.268725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:46.999303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268895+0000","last_change":"2024-03-16T14:31:06.704882+0000","last_active":"2024-03-16T14:31:45.268895+0000","last_peered":"2024-03-16T14:31:45.268895+0000","last_clean":"2024-03-16T14:31:45.268895+0000","last_became_active":"2024-03-16T14:31:06.704497+0000","last_became_peered":"2024-03-16T14:31:06.704497+0000","last_unstale":"2024-03-16T14:31:45.268895+0000","last_undegraded":"2024-03-16T14:31:45.268895+0000","last_fullsized":"2024-03-16T14:31:45.268895+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:01.825949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128632+0000","last_change":"2024-03-16T14:31:10.722335+0000","last_active":"2024-03-16T14:31:45.128632+0000","last_peered":"2024-03-16T14:31:45.128632+0000","last_clean":"2024-03-16T14:31:45.128632+0000","last_became_active":"2024-03-16T14:31:10.721682+0000","last_became_peered":"2024-03-16T14:31:10.721682+0000","last_unstale":"2024-03-16T14:31:45.128632+0000","last_undegraded":"2024-03-16T14:31:45.128632+0000","last_fullsized":"2024-03-16T14:31:45.128632+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:06.813691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268325+0000","last_change":"2024-03-16T14:31:08.714189+0000","last_active":"2024-03-16T14:31:45.268325+0000","last_peered":"2024-03-16T14:31:45.268325+0000","last_clean":"2024-03-16T14:31:45.268325+0000","last_became_active":"2024-03-16T14:31:08.713878+0000","last_became_peered":"2024-03-16T14:31:08.713878+0000","last_unstale":"2024-03-16T14:31:45.268325+0000","last_undegraded":"2024-03-16T14:31:45.268325+0000","last_fullsized":"2024-03-16T14:31:45.268325+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:28:06.729052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008775+0000","last_change":"2024-03-16T14:31:04.671108+0000","last_active":"2024-03-16T14:31:45.008775+0000","last_peered":"2024-03-16T14:31:45.008775+0000","last_clean":"2024-03-16T14:31:45.008775+0000","last_became_active":"2024-03-16T14:31:04.670757+0000","last_became_peered":"2024-03-16T14:31:04.670757+0000","last_unstale":"2024-03-16T14:31:45.008775+0000","last_undegraded":"2024-03-16T14:31:45.008775+0000","last_fullsized":"2024-03-16T14:31:45.008775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:49:15.942210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268859+0000","last_change":"2024-03-16T14:31:06.699333+0000","last_active":"2024-03-16T14:31:45.268859+0000","last_peered":"2024-03-16T14:31:45.268859+0000","last_clean":"2024-03-16T14:31:45.268859+0000","last_became_active":"2024-03-16T14:31:06.698971+0000","last_became_peered":"2024-03-16T14:31:06.698971+0000","last_unstale":"2024-03-16T14:31:45.268859+0000","last_undegraded":"2024-03-16T14:31:45.268859+0000","last_fullsized":"2024-03-16T14:31:45.268859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:05:17.236078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269917+0000","last_change":"2024-03-16T14:31:11.205614+0000","last_active":"2024-03-16T14:31:45.269917+0000","last_peered":"2024-03-16T14:31:45.269917+0000","last_clean":"2024-03-16T14:31:45.269917+0000","last_became_active":"2024-03-16T14:31:11.205274+0000","last_became_peered":"2024-03-16T14:31:11.205274+0000","last_unstale":"2024-03-16T14:31:45.269917+0000","last_undegraded":"2024-03-16T14:31:45.269917+0000","last_fullsized":"2024-03-16T14:31:45.269917+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:11.173148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128829+0000","last_change":"2024-03-16T14:31:08.708811+0000","last_active":"2024-03-16T14:31:45.128829+0000","last_peered":"2024-03-16T14:31:45.128829+0000","last_clean":"2024-03-16T14:31:45.128829+0000","last_became_active":"2024-03-16T14:31:08.708487+0000","last_became_peered":"2024-03-16T14:31:08.708487+0000","last_unstale":"2024-03-16T14:31:45.128829+0000","last_undegraded":"2024-03-16T14:31:45.128829+0000","last_fullsized":"2024-03-16T14:31:45.128829+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:58:10.235815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114106+0000","last_change":"2024-03-16T14:31:04.674161+0000","last_active":"2024-03-16T14:31:45.114106+0000","last_peered":"2024-03-16T14:31:45.114106+0000","last_clean":"2024-03-16T14:31:45.114106+0000","last_became_active":"2024-03-16T14:31:04.673858+0000","last_became_peered":"2024-03-16T14:31:04.673858+0000","last_unstale":"2024-03-16T14:31:45.114106+0000","last_undegraded":"2024-03-16T14:31:45.114106+0000","last_fullsized":"2024-03-16T14:31:45.114106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:49:02.685654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269569+0000","last_change":"2024-03-16T14:31:06.698174+0000","last_active":"2024-03-16T14:31:45.269569+0000","last_peered":"2024-03-16T14:31:45.269569+0000","last_clean":"2024-03-16T14:31:45.269569+0000","last_became_active":"2024-03-16T14:31:06.697855+0000","last_became_peered":"2024-03-16T14:31:06.697855+0000","last_unstale":"2024-03-16T14:31:45.269569+0000","last_undegraded":"2024-03-16T14:31:45.269569+0000","last_fullsized":"2024-03-16T14:31:45.269569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:03.834268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008216+0000","last_change":"2024-03-16T14:31:10.719826+0000","last_active":"2024-03-16T14:31:45.008216+0000","last_peered":"2024-03-16T14:31:45.008216+0000","last_clean":"2024-03-16T14:31:45.008216+0000","last_became_active":"2024-03-16T14:31:10.719059+0000","last_became_peered":"2024-03-16T14:31:10.719059+0000","last_unstale":"2024-03-16T14:31:45.008216+0000","last_undegraded":"2024-03-16T14:31:45.008216+0000","last_fullsized":"2024-03-16T14:31:45.008216+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:32.526482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270192+0000","last_change":"2024-03-16T14:31:08.707185+0000","last_active":"2024-03-16T14:31:45.270192+0000","last_peered":"2024-03-16T14:31:45.270192+0000","last_clean":"2024-03-16T14:31:45.270192+0000","last_became_active":"2024-03-16T14:31:08.706857+0000","last_became_peered":"2024-03-16T14:31:08.706857+0000","last_unstale":"2024-03-16T14:31:45.270192+0000","last_undegraded":"2024-03-16T14:31:45.270192+0000","last_fullsized":"2024-03-16T14:31:45.270192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:51:18.427073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128915+0000","last_change":"2024-03-16T14:31:04.677626+0000","last_active":"2024-03-16T14:31:45.128915+0000","last_peered":"2024-03-16T14:31:45.128915+0000","last_clean":"2024-03-16T14:31:45.128915+0000","last_became_active":"2024-03-16T14:31:04.677326+0000","last_became_peered":"2024-03-16T14:31:04.677326+0000","last_unstale":"2024-03-16T14:31:45.128915+0000","last_undegraded":"2024-03-16T14:31:45.128915+0000","last_fullsized":"2024-03-16T14:31:45.128915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:08:14.371875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269655+0000","last_change":"2024-03-16T14:31:06.698345+0000","last_active":"2024-03-16T14:31:45.269655+0000","last_peered":"2024-03-16T14:31:45.269655+0000","last_clean":"2024-03-16T14:31:45.269655+0000","last_became_active":"2024-03-16T14:31:06.697407+0000","last_became_peered":"2024-03-16T14:31:06.697407+0000","last_unstale":"2024-03-16T14:31:45.269655+0000","last_undegraded":"2024-03-16T14:31:45.269655+0000","last_fullsized":"2024-03-16T14:31:45.269655+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:20:24.994698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268976+0000","last_change":"2024-03-16T14:31:10.722774+0000","last_active":"2024-03-16T14:31:45.268976+0000","last_peered":"2024-03-16T14:31:45.268976+0000","last_clean":"2024-03-16T14:31:45.268976+0000","last_became_active":"2024-03-16T14:31:10.722401+0000","last_became_peered":"2024-03-16T14:31:10.722401+0000","last_unstale":"2024-03-16T14:31:45.268976+0000","last_undegraded":"2024-03-16T14:31:45.268976+0000","last_fullsized":"2024-03-16T14:31:45.268976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:38:55.215778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269416+0000","last_change":"2024-03-16T14:31:08.714961+0000","last_active":"2024-03-16T14:31:45.269416+0000","last_peered":"2024-03-16T14:31:45.269416+0000","last_clean":"2024-03-16T14:31:45.269416+0000","last_became_active":"2024-03-16T14:31:08.713755+0000","last_became_peered":"2024-03-16T14:31:08.713755+0000","last_unstale":"2024-03-16T14:31:45.269416+0000","last_undegraded":"2024-03-16T14:31:45.269416+0000","last_fullsized":"2024-03-16T14:31:45.269416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:47:31.213900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010185+0000","last_change":"2024-03-16T14:31:04.673573+0000","last_active":"2024-03-16T14:31:45.010185+0000","last_peered":"2024-03-16T14:31:45.010185+0000","last_clean":"2024-03-16T14:31:45.010185+0000","last_became_active":"2024-03-16T14:31:04.673201+0000","last_became_peered":"2024-03-16T14:31:04.673201+0000","last_unstale":"2024-03-16T14:31:45.010185+0000","last_undegraded":"2024-03-16T14:31:45.010185+0000","last_fullsized":"2024-03-16T14:31:45.010185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:22:08.408440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009681+0000","last_change":"2024-03-16T14:31:06.700955+0000","last_active":"2024-03-16T14:31:45.009681+0000","last_peered":"2024-03-16T14:31:45.009681+0000","last_clean":"2024-03-16T14:31:45.009681+0000","last_became_active":"2024-03-16T14:31:06.700681+0000","last_became_peered":"2024-03-16T14:31:06.700681+0000","last_unstale":"2024-03-16T14:31:45.009681+0000","last_undegraded":"2024-03-16T14:31:45.009681+0000","last_fullsized":"2024-03-16T14:31:45.009681+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:17:34.317137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010919+0000","last_change":"2024-03-16T14:31:10.726683+0000","last_active":"2024-03-16T14:31:45.010919+0000","last_peered":"2024-03-16T14:31:45.010919+0000","last_clean":"2024-03-16T14:31:45.010919+0000","last_became_active":"2024-03-16T14:31:10.726274+0000","last_became_peered":"2024-03-16T14:31:10.726274+0000","last_unstale":"2024-03-16T14:31:45.010919+0000","last_undegraded":"2024-03-16T14:31:45.010919+0000","last_fullsized":"2024-03-16T14:31:45.010919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:24:43.244002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734560+0000","last_change":"2024-03-16T14:31:08.706557+0000","last_active":"2024-03-16T14:32:19.734560+0000","last_peered":"2024-03-16T14:32:19.734560+0000","last_clean":"2024-03-16T14:32:19.734560+0000","last_became_active":"2024-03-16T14:31:08.706200+0000","last_became_peered":"2024-03-16T14:31:08.706200+0000","last_unstale":"2024-03-16T14:32:19.734560+0000","last_undegraded":"2024-03-16T14:32:19.734560+0000","last_fullsized":"2024-03-16T14:32:19.734560+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:38:43.435022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128759+0000","last_change":"2024-03-16T14:31:04.675942+0000","last_active":"2024-03-16T14:31:45.128759+0000","last_peered":"2024-03-16T14:31:45.128759+0000","last_clean":"2024-03-16T14:31:45.128759+0000","last_became_active":"2024-03-16T14:31:04.675663+0000","last_became_peered":"2024-03-16T14:31:04.675663+0000","last_unstale":"2024-03-16T14:31:45.128759+0000","last_undegraded":"2024-03-16T14:31:45.128759+0000","last_fullsized":"2024-03-16T14:31:45.128759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:02.772572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269546+0000","last_change":"2024-03-16T14:31:06.691744+0000","last_active":"2024-03-16T14:31:45.269546+0000","last_peered":"2024-03-16T14:31:45.269546+0000","last_clean":"2024-03-16T14:31:45.269546+0000","last_became_active":"2024-03-16T14:31:06.691382+0000","last_became_peered":"2024-03-16T14:31:06.691382+0000","last_unstale":"2024-03-16T14:31:45.269546+0000","last_undegraded":"2024-03-16T14:31:45.269546+0000","last_fullsized":"2024-03-16T14:31:45.269546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:06.922360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010661+0000","last_change":"2024-03-16T14:31:10.718741+0000","last_active":"2024-03-16T14:31:45.010661+0000","last_peered":"2024-03-16T14:31:45.010661+0000","last_clean":"2024-03-16T14:31:45.010661+0000","last_became_active":"2024-03-16T14:31:10.718431+0000","last_became_peered":"2024-03-16T14:31:10.718431+0000","last_unstale":"2024-03-16T14:31:45.010661+0000","last_undegraded":"2024-03-16T14:31:45.010661+0000","last_fullsized":"2024-03-16T14:31:45.010661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:31:52.312816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734793+0000","last_change":"2024-03-16T14:31:08.705967+0000","last_active":"2024-03-16T14:32:19.734793+0000","last_peered":"2024-03-16T14:32:19.734793+0000","last_clean":"2024-03-16T14:32:19.734793+0000","last_became_active":"2024-03-16T14:31:08.705648+0000","last_became_peered":"2024-03-16T14:31:08.705648+0000","last_unstale":"2024-03-16T14:32:19.734793+0000","last_undegraded":"2024-03-16T14:32:19.734793+0000","last_fullsized":"2024-03-16T14:32:19.734793+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:59.643938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010830+0000","last_change":"2024-03-16T14:31:04.674328+0000","last_active":"2024-03-16T14:31:45.010830+0000","last_peered":"2024-03-16T14:31:45.010830+0000","last_clean":"2024-03-16T14:31:45.010830+0000","last_became_active":"2024-03-16T14:31:04.673998+0000","last_became_peered":"2024-03-16T14:31:04.673998+0000","last_unstale":"2024-03-16T14:31:45.010830+0000","last_undegraded":"2024-03-16T14:31:45.010830+0000","last_fullsized":"2024-03-16T14:31:45.010830+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:30:39.481993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114379+0000","last_change":"2024-03-16T14:31:06.703133+0000","last_active":"2024-03-16T14:31:45.114379+0000","last_peered":"2024-03-16T14:31:45.114379+0000","last_clean":"2024-03-16T14:31:45.114379+0000","last_became_active":"2024-03-16T14:31:06.702533+0000","last_became_peered":"2024-03-16T14:31:06.702533+0000","last_unstale":"2024-03-16T14:31:45.114379+0000","last_undegraded":"2024-03-16T14:31:45.114379+0000","last_fullsized":"2024-03-16T14:31:45.114379+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:06.809872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.122340+0000","last_change":"2024-03-16T14:31:10.721796+0000","last_active":"2024-03-16T14:31:45.122340+0000","last_peered":"2024-03-16T14:31:45.122340+0000","last_clean":"2024-03-16T14:31:45.122340+0000","last_became_active":"2024-03-16T14:31:10.721486+0000","last_became_peered":"2024-03-16T14:31:10.721486+0000","last_unstale":"2024-03-16T14:31:45.122340+0000","last_undegraded":"2024-03-16T14:31:45.122340+0000","last_fullsized":"2024-03-16T14:31:45.122340+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:12:42.239455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270861+0000","last_change":"2024-03-16T14:31:08.708164+0000","last_active":"2024-03-16T14:31:45.270861+0000","last_peered":"2024-03-16T14:31:45.270861+0000","last_clean":"2024-03-16T14:31:45.270861+0000","last_became_active":"2024-03-16T14:31:08.707311+0000","last_became_peered":"2024-03-16T14:31:08.707311+0000","last_unstale":"2024-03-16T14:31:45.270861+0000","last_undegraded":"2024-03-16T14:31:45.270861+0000","last_fullsized":"2024-03-16T14:31:45.270861+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:24:35.988978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128660+0000","last_change":"2024-03-16T14:31:04.677705+0000","last_active":"2024-03-16T14:31:45.128660+0000","last_peered":"2024-03-16T14:31:45.128660+0000","last_clean":"2024-03-16T14:31:45.128660+0000","last_became_active":"2024-03-16T14:31:04.677379+0000","last_became_peered":"2024-03-16T14:31:04.677379+0000","last_unstale":"2024-03-16T14:31:45.128660+0000","last_undegraded":"2024-03-16T14:31:45.128660+0000","last_fullsized":"2024-03-16T14:31:45.128660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:28.257779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011380+0000","last_change":"2024-03-16T14:31:06.702706+0000","last_active":"2024-03-16T14:31:45.011380+0000","last_peered":"2024-03-16T14:31:45.011380+0000","last_clean":"2024-03-16T14:31:45.011380+0000","last_became_active":"2024-03-16T14:31:06.702377+0000","last_became_peered":"2024-03-16T14:31:06.702377+0000","last_unstale":"2024-03-16T14:31:45.011380+0000","last_undegraded":"2024-03-16T14:31:45.011380+0000","last_fullsized":"2024-03-16T14:31:45.011380+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:22.020315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128921+0000","last_change":"2024-03-16T14:31:11.204053+0000","last_active":"2024-03-16T14:31:45.128921+0000","last_peered":"2024-03-16T14:31:45.128921+0000","last_clean":"2024-03-16T14:31:45.128921+0000","last_became_active":"2024-03-16T14:31:11.203738+0000","last_became_peered":"2024-03-16T14:31:11.203738+0000","last_unstale":"2024-03-16T14:31:45.128921+0000","last_undegraded":"2024-03-16T14:31:45.128921+0000","last_fullsized":"2024-03-16T14:31:45.128921+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:28:18.748789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114131+0000","last_change":"2024-03-16T14:31:08.709074+0000","last_active":"2024-03-16T14:31:45.114131+0000","last_peered":"2024-03-16T14:31:45.114131+0000","last_clean":"2024-03-16T14:31:45.114131+0000","last_became_active":"2024-03-16T14:31:08.708778+0000","last_became_peered":"2024-03-16T14:31:08.708778+0000","last_unstale":"2024-03-16T14:31:45.114131+0000","last_undegraded":"2024-03-16T14:31:45.114131+0000","last_fullsized":"2024-03-16T14:31:45.114131+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:36.373710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268271+0000","last_change":"2024-03-16T14:31:04.668540+0000","last_active":"2024-03-16T14:31:45.268271+0000","last_peered":"2024-03-16T14:31:45.268271+0000","last_clean":"2024-03-16T14:31:45.268271+0000","last_became_active":"2024-03-16T14:31:04.668221+0000","last_became_peered":"2024-03-16T14:31:04.668221+0000","last_unstale":"2024-03-16T14:31:45.268271+0000","last_undegraded":"2024-03-16T14:31:45.268271+0000","last_fullsized":"2024-03-16T14:31:45.268271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:10:00.925985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270966+0000","last_change":"2024-03-16T14:31:06.696810+0000","last_active":"2024-03-16T14:31:45.270966+0000","last_peered":"2024-03-16T14:31:45.270966+0000","last_clean":"2024-03-16T14:31:45.270966+0000","last_became_active":"2024-03-16T14:31:06.696459+0000","last_became_peered":"2024-03-16T14:31:06.696459+0000","last_unstale":"2024-03-16T14:31:45.270966+0000","last_undegraded":"2024-03-16T14:31:45.270966+0000","last_fullsized":"2024-03-16T14:31:45.270966+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:11.615069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114187+0000","last_change":"2024-03-16T14:31:11.202128+0000","last_active":"2024-03-16T14:31:45.114187+0000","last_peered":"2024-03-16T14:31:45.114187+0000","last_clean":"2024-03-16T14:31:45.114187+0000","last_became_active":"2024-03-16T14:31:11.201786+0000","last_became_peered":"2024-03-16T14:31:11.201786+0000","last_unstale":"2024-03-16T14:31:45.114187+0000","last_undegraded":"2024-03-16T14:31:45.114187+0000","last_fullsized":"2024-03-16T14:31:45.114187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:29.379451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129172+0000","last_change":"2024-03-16T14:31:04.679710+0000","last_active":"2024-03-16T14:31:45.129172+0000","last_peered":"2024-03-16T14:31:45.129172+0000","last_clean":"2024-03-16T14:31:45.129172+0000","last_became_active":"2024-03-16T14:31:04.679433+0000","last_became_peered":"2024-03-16T14:31:04.679433+0000","last_unstale":"2024-03-16T14:31:45.129172+0000","last_undegraded":"2024-03-16T14:31:45.129172+0000","last_fullsized":"2024-03-16T14:31:45.129172+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:27.002679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113701+0000","last_change":"2024-03-16T14:31:06.701063+0000","last_active":"2024-03-16T14:31:45.113701+0000","last_peered":"2024-03-16T14:31:45.113701+0000","last_clean":"2024-03-16T14:31:45.113701+0000","last_became_active":"2024-03-16T14:31:06.700738+0000","last_became_peered":"2024-03-16T14:31:06.700738+0000","last_unstale":"2024-03-16T14:31:45.113701+0000","last_undegraded":"2024-03-16T14:31:45.113701+0000","last_fullsized":"2024-03-16T14:31:45.113701+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:57:13.931074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734560+0000","last_change":"2024-03-16T14:31:08.708111+0000","last_active":"2024-03-16T14:32:19.734560+0000","last_peered":"2024-03-16T14:32:19.734560+0000","last_clean":"2024-03-16T14:32:19.734560+0000","last_became_active":"2024-03-16T14:31:08.707770+0000","last_became_peered":"2024-03-16T14:31:08.707770+0000","last_unstale":"2024-03-16T14:32:19.734560+0000","last_undegraded":"2024-03-16T14:32:19.734560+0000","last_fullsized":"2024-03-16T14:32:19.734560+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:18.898584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270011+0000","last_change":"2024-03-16T14:31:10.724541+0000","last_active":"2024-03-16T14:31:45.270011+0000","last_peered":"2024-03-16T14:31:45.270011+0000","last_clean":"2024-03-16T14:31:45.270011+0000","last_became_active":"2024-03-16T14:31:10.724142+0000","last_became_peered":"2024-03-16T14:31:10.724142+0000","last_unstale":"2024-03-16T14:31:45.270011+0000","last_undegraded":"2024-03-16T14:31:45.270011+0000","last_fullsized":"2024-03-16T14:31:45.270011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:03:35.534603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129121+0000","last_change":"2024-03-16T14:31:04.678562+0000","last_active":"2024-03-16T14:31:45.129121+0000","last_peered":"2024-03-16T14:31:45.129121+0000","last_clean":"2024-03-16T14:31:45.129121+0000","last_became_active":"2024-03-16T14:31:04.678266+0000","last_became_peered":"2024-03-16T14:31:04.678266+0000","last_unstale":"2024-03-16T14:31:45.129121+0000","last_undegraded":"2024-03-16T14:31:45.129121+0000","last_fullsized":"2024-03-16T14:31:45.129121+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:44.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008308+0000","last_change":"2024-03-16T14:31:06.704081+0000","last_active":"2024-03-16T14:31:45.008308+0000","last_peered":"2024-03-16T14:31:45.008308+0000","last_clean":"2024-03-16T14:31:45.008308+0000","last_became_active":"2024-03-16T14:31:06.703762+0000","last_became_peered":"2024-03-16T14:31:06.703762+0000","last_unstale":"2024-03-16T14:31:45.008308+0000","last_undegraded":"2024-03-16T14:31:45.008308+0000","last_fullsized":"2024-03-16T14:31:45.008308+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:47.127996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121293+0000","last_change":"2024-03-16T14:31:08.710814+0000","last_active":"2024-03-16T14:31:45.121293+0000","last_peered":"2024-03-16T14:31:45.121293+0000","last_clean":"2024-03-16T14:31:45.121293+0000","last_became_active":"2024-03-16T14:31:08.710314+0000","last_became_peered":"2024-03-16T14:31:08.710314+0000","last_unstale":"2024-03-16T14:31:45.121293+0000","last_undegraded":"2024-03-16T14:31:45.121293+0000","last_fullsized":"2024-03-16T14:31:45.121293+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:53:31.627429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128467+0000","last_change":"2024-03-16T14:31:10.725143+0000","last_active":"2024-03-16T14:31:45.128467+0000","last_peered":"2024-03-16T14:31:45.128467+0000","last_clean":"2024-03-16T14:31:45.128467+0000","last_became_active":"2024-03-16T14:31:10.724860+0000","last_became_peered":"2024-03-16T14:31:10.724860+0000","last_unstale":"2024-03-16T14:31:45.128467+0000","last_undegraded":"2024-03-16T14:31:45.128467+0000","last_fullsized":"2024-03-16T14:31:45.128467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:09.584785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121552+0000","last_change":"2024-03-16T14:31:08.707272+0000","last_active":"2024-03-16T14:31:45.121552+0000","last_peered":"2024-03-16T14:31:45.121552+0000","last_clean":"2024-03-16T14:31:45.121552+0000","last_became_active":"2024-03-16T14:31:08.706314+0000","last_became_peered":"2024-03-16T14:31:08.706314+0000","last_unstale":"2024-03-16T14:31:45.121552+0000","last_undegraded":"2024-03-16T14:31:45.121552+0000","last_fullsized":"2024-03-16T14:31:45.121552+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:34.998482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129298+0000","last_change":"2024-03-16T14:31:04.672399+0000","last_active":"2024-03-16T14:31:45.129298+0000","last_peered":"2024-03-16T14:31:45.129298+0000","last_clean":"2024-03-16T14:31:45.129298+0000","last_became_active":"2024-03-16T14:31:04.672092+0000","last_became_peered":"2024-03-16T14:31:04.672092+0000","last_unstale":"2024-03-16T14:31:45.129298+0000","last_undegraded":"2024-03-16T14:31:45.129298+0000","last_fullsized":"2024-03-16T14:31:45.129298+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:55.442342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268696+0000","last_change":"2024-03-16T14:31:06.705167+0000","last_active":"2024-03-16T14:31:45.268696+0000","last_peered":"2024-03-16T14:31:45.268696+0000","last_clean":"2024-03-16T14:31:45.268696+0000","last_became_active":"2024-03-16T14:31:06.704852+0000","last_became_peered":"2024-03-16T14:31:06.704852+0000","last_unstale":"2024-03-16T14:31:45.268696+0000","last_undegraded":"2024-03-16T14:31:45.268696+0000","last_fullsized":"2024-03-16T14:31:45.268696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:22.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008482+0000","last_change":"2024-03-16T14:31:10.721287+0000","last_active":"2024-03-16T14:31:45.008482+0000","last_peered":"2024-03-16T14:31:45.008482+0000","last_clean":"2024-03-16T14:31:45.008482+0000","last_became_active":"2024-03-16T14:31:10.720939+0000","last_became_peered":"2024-03-16T14:31:10.720939+0000","last_unstale":"2024-03-16T14:31:45.008482+0000","last_undegraded":"2024-03-16T14:31:45.008482+0000","last_fullsized":"2024-03-16T14:31:45.008482+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:00.942919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121461+0000","last_change":"2024-03-16T14:31:08.708168+0000","last_active":"2024-03-16T14:31:45.121461+0000","last_peered":"2024-03-16T14:31:45.121461+0000","last_clean":"2024-03-16T14:31:45.121461+0000","last_became_active":"2024-03-16T14:31:08.707780+0000","last_became_peered":"2024-03-16T14:31:08.707780+0000","last_unstale":"2024-03-16T14:31:45.121461+0000","last_undegraded":"2024-03-16T14:31:45.121461+0000","last_fullsized":"2024-03-16T14:31:45.121461+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:33.833906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114023+0000","last_change":"2024-03-16T14:31:04.670724+0000","last_active":"2024-03-16T14:31:45.114023+0000","last_peered":"2024-03-16T14:31:45.114023+0000","last_clean":"2024-03-16T14:31:45.114023+0000","last_became_active":"2024-03-16T14:31:04.670432+0000","last_became_peered":"2024-03-16T14:31:04.670432+0000","last_unstale":"2024-03-16T14:31:45.114023+0000","last_undegraded":"2024-03-16T14:31:45.114023+0000","last_fullsized":"2024-03-16T14:31:45.114023+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:53:51.765443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011367+0000","last_change":"2024-03-16T14:31:06.698773+0000","last_active":"2024-03-16T14:31:45.011367+0000","last_peered":"2024-03-16T14:31:45.011367+0000","last_clean":"2024-03-16T14:31:45.011367+0000","last_became_active":"2024-03-16T14:31:06.698472+0000","last_became_peered":"2024-03-16T14:31:06.698472+0000","last_unstale":"2024-03-16T14:31:45.011367+0000","last_undegraded":"2024-03-16T14:31:45.011367+0000","last_fullsized":"2024-03-16T14:31:45.011367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:48.174804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270468+0000","last_change":"2024-03-16T14:31:10.719141+0000","last_active":"2024-03-16T14:31:45.270468+0000","last_peered":"2024-03-16T14:31:45.270468+0000","last_clean":"2024-03-16T14:31:45.270468+0000","last_became_active":"2024-03-16T14:31:10.718652+0000","last_became_peered":"2024-03-16T14:31:10.718652+0000","last_unstale":"2024-03-16T14:31:45.270468+0000","last_undegraded":"2024-03-16T14:31:45.270468+0000","last_fullsized":"2024-03-16T14:31:45.270468+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:08.345560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113804+0000","last_change":"2024-03-16T14:31:08.701635+0000","last_active":"2024-03-16T14:31:45.113804+0000","last_peered":"2024-03-16T14:31:45.113804+0000","last_clean":"2024-03-16T14:31:45.113804+0000","last_became_active":"2024-03-16T14:31:04.674571+0000","last_became_peered":"2024-03-16T14:31:04.674571+0000","last_unstale":"2024-03-16T14:31:45.113804+0000","last_undegraded":"2024-03-16T14:31:45.113804+0000","last_fullsized":"2024-03-16T14:31:45.113804+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:41.218336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012146240000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008879+0000","last_change":"2024-03-16T14:31:04.669970+0000","last_active":"2024-03-16T14:31:45.008879+0000","last_peered":"2024-03-16T14:31:45.008879+0000","last_clean":"2024-03-16T14:31:45.008879+0000","last_became_active":"2024-03-16T14:31:04.669674+0000","last_became_peered":"2024-03-16T14:31:04.669674+0000","last_unstale":"2024-03-16T14:31:45.008879+0000","last_undegraded":"2024-03-16T14:31:45.008879+0000","last_fullsized":"2024-03-16T14:31:45.008879+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:54.842931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":94,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734518+0000","last_change":"2024-03-16T14:31:06.693402+0000","last_active":"2024-03-16T14:32:19.734518+0000","last_peered":"2024-03-16T14:32:19.734518+0000","last_clean":"2024-03-16T14:32:19.734518+0000","last_became_active":"2024-03-16T14:31:06.693065+0000","last_became_peered":"2024-03-16T14:31:06.693065+0000","last_unstale":"2024-03-16T14:32:19.734518+0000","last_undegraded":"2024-03-16T14:32:19.734518+0000","last_fullsized":"2024-03-16T14:32:19.734518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:54.615938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010780+0000","last_change":"2024-03-16T14:31:08.714945+0000","last_active":"2024-03-16T14:31:45.010780+0000","last_peered":"2024-03-16T14:31:45.010780+0000","last_clean":"2024-03-16T14:31:45.010780+0000","last_became_active":"2024-03-16T14:31:08.714637+0000","last_became_peered":"2024-03-16T14:31:08.714637+0000","last_unstale":"2024-03-16T14:31:45.010780+0000","last_undegraded":"2024-03-16T14:31:45.010780+0000","last_fullsized":"2024-03-16T14:31:45.010780+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:45:31.769929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129354+0000","last_change":"2024-03-16T14:31:11.202130+0000","last_active":"2024-03-16T14:31:45.129354+0000","last_peered":"2024-03-16T14:31:45.129354+0000","last_clean":"2024-03-16T14:31:45.129354+0000","last_became_active":"2024-03-16T14:31:11.201832+0000","last_became_peered":"2024-03-16T14:31:11.201832+0000","last_unstale":"2024-03-16T14:31:45.129354+0000","last_undegraded":"2024-03-16T14:31:45.129354+0000","last_fullsized":"2024-03-16T14:31:45.129354+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:11.276429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268713+0000","last_change":"2024-03-16T14:31:04.679021+0000","last_active":"2024-03-16T14:31:45.268713+0000","last_peered":"2024-03-16T14:31:45.268713+0000","last_clean":"2024-03-16T14:31:45.268713+0000","last_became_active":"2024-03-16T14:31:04.678786+0000","last_became_peered":"2024-03-16T14:31:04.678786+0000","last_unstale":"2024-03-16T14:31:45.268713+0000","last_undegraded":"2024-03-16T14:31:45.268713+0000","last_fullsized":"2024-03-16T14:31:45.268713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:22.898738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121623+0000","last_change":"2024-03-16T14:31:08.695385+0000","last_active":"2024-03-16T14:31:45.121623+0000","last_peered":"2024-03-16T14:31:45.121623+0000","last_clean":"2024-03-16T14:31:45.121623+0000","last_became_active":"2024-03-16T14:31:04.671086+0000","last_became_peered":"2024-03-16T14:31:04.671086+0000","last_unstale":"2024-03-16T14:31:45.121623+0000","last_undegraded":"2024-03-16T14:31:45.121623+0000","last_fullsized":"2024-03-16T14:31:45.121623+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:23.153102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077491400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270395+0000","last_change":"2024-03-16T14:31:06.697643+0000","last_active":"2024-03-16T14:31:45.270395+0000","last_peered":"2024-03-16T14:31:45.270395+0000","last_clean":"2024-03-16T14:31:45.270395+0000","last_became_active":"2024-03-16T14:31:06.696749+0000","last_became_peered":"2024-03-16T14:31:06.696749+0000","last_unstale":"2024-03-16T14:31:45.270395+0000","last_undegraded":"2024-03-16T14:31:45.270395+0000","last_fullsized":"2024-03-16T14:31:45.270395+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:55.278292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121950+0000","last_change":"2024-03-16T14:31:08.709870+0000","last_active":"2024-03-16T14:31:45.121950+0000","last_peered":"2024-03-16T14:31:45.121950+0000","last_clean":"2024-03-16T14:31:45.121950+0000","last_became_active":"2024-03-16T14:31:08.709533+0000","last_became_peered":"2024-03-16T14:31:08.709533+0000","last_unstale":"2024-03-16T14:31:45.121950+0000","last_undegraded":"2024-03-16T14:31:45.121950+0000","last_fullsized":"2024-03-16T14:31:45.121950+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:08.155629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114327+0000","last_change":"2024-03-16T14:31:10.724387+0000","last_active":"2024-03-16T14:31:45.114327+0000","last_peered":"2024-03-16T14:31:45.114327+0000","last_clean":"2024-03-16T14:31:45.114327+0000","last_became_active":"2024-03-16T14:31:10.724048+0000","last_became_peered":"2024-03-16T14:31:10.724048+0000","last_unstale":"2024-03-16T14:31:45.114327+0000","last_undegraded":"2024-03-16T14:31:45.114327+0000","last_fullsized":"2024-03-16T14:31:45.114327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:26.403169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":345,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:47.176163+0000","last_change":"2024-03-16T14:30:55.235452+0000","last_active":"2024-03-16T14:31:47.176163+0000","last_peered":"2024-03-16T14:31:47.176163+0000","last_clean":"2024-03-16T14:31:47.176163+0000","last_became_active":"2024-03-16T14:30:55.235047+0000","last_became_peered":"2024-03-16T14:30:55.235047+0000","last_unstale":"2024-03-16T14:31:47.176163+0000","last_undegraded":"2024-03-16T14:31:47.176163+0000","last_fullsized":"2024-03-16T14:31:47.176163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:28:44.874450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:28:44.874450+0000","last_clean_scrub_stamp":"2024-03-16T14:28:44.874450+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:13.723888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269239+0000","last_change":"2024-03-16T14:31:04.670629+0000","last_active":"2024-03-16T14:31:45.269239+0000","last_peered":"2024-03-16T14:31:45.269239+0000","last_clean":"2024-03-16T14:31:45.269239+0000","last_became_active":"2024-03-16T14:31:04.670344+0000","last_became_peered":"2024-03-16T14:31:04.670344+0000","last_unstale":"2024-03-16T14:31:45.269239+0000","last_undegraded":"2024-03-16T14:31:45.269239+0000","last_fullsized":"2024-03-16T14:31:45.269239+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:41:48.580850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734577+0000","last_change":"2024-03-16T14:31:06.698898+0000","last_active":"2024-03-16T14:32:19.734577+0000","last_peered":"2024-03-16T14:32:19.734577+0000","last_clean":"2024-03-16T14:32:19.734577+0000","last_became_active":"2024-03-16T14:31:06.698386+0000","last_became_peered":"2024-03-16T14:31:06.698386+0000","last_unstale":"2024-03-16T14:32:19.734577+0000","last_undegraded":"2024-03-16T14:32:19.734577+0000","last_fullsized":"2024-03-16T14:32:19.734577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:20:54.053320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129026+0000","last_change":"2024-03-16T14:31:08.714519+0000","last_active":"2024-03-16T14:31:45.129026+0000","last_peered":"2024-03-16T14:31:45.129026+0000","last_clean":"2024-03-16T14:31:45.129026+0000","last_became_active":"2024-03-16T14:31:08.713445+0000","last_became_peered":"2024-03-16T14:31:08.713445+0000","last_unstale":"2024-03-16T14:31:45.129026+0000","last_undegraded":"2024-03-16T14:31:45.129026+0000","last_fullsized":"2024-03-16T14:31:45.129026+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:50.671228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269772+0000","last_change":"2024-03-16T14:31:10.720846+0000","last_active":"2024-03-16T14:31:45.269772+0000","last_peered":"2024-03-16T14:31:45.269772+0000","last_clean":"2024-03-16T14:31:45.269772+0000","last_became_active":"2024-03-16T14:31:10.720513+0000","last_became_peered":"2024-03-16T14:31:10.720513+0000","last_unstale":"2024-03-16T14:31:45.269772+0000","last_undegraded":"2024-03-16T14:31:45.269772+0000","last_fullsized":"2024-03-16T14:31:45.269772+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:02:26.456923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008708+0000","last_change":"2024-03-16T14:31:04.664536+0000","last_active":"2024-03-16T14:31:45.008708+0000","last_peered":"2024-03-16T14:31:45.008708+0000","last_clean":"2024-03-16T14:31:45.008708+0000","last_became_active":"2024-03-16T14:31:04.664242+0000","last_became_peered":"2024-03-16T14:31:04.664242+0000","last_unstale":"2024-03-16T14:31:45.008708+0000","last_undegraded":"2024-03-16T14:31:45.008708+0000","last_fullsized":"2024-03-16T14:31:45.008708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:01:05.142473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":118,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:24.778005+0000","last_change":"2024-03-16T14:31:08.701212+0000","last_active":"2024-03-16T14:32:24.778005+0000","last_peered":"2024-03-16T14:32:24.778005+0000","last_clean":"2024-03-16T14:32:24.778005+0000","last_became_active":"2024-03-16T14:31:04.680416+0000","last_became_peered":"2024-03-16T14:31:04.680416+0000","last_unstale":"2024-03-16T14:32:24.778005+0000","last_undegraded":"2024-03-16T14:32:24.778005+0000","last_fullsized":"2024-03-16T14:32:24.778005+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:48:48.659045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057932800000000005,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011021+0000","last_change":"2024-03-16T14:31:32.535936+0000","last_active":"2024-03-16T14:31:45.011021+0000","last_peered":"2024-03-16T14:31:45.011021+0000","last_clean":"2024-03-16T14:31:45.011021+0000","last_became_active":"2024-03-16T14:31:32.528406+0000","last_became_peered":"2024-03-16T14:31:32.528406+0000","last_unstale":"2024-03-16T14:31:45.011021+0000","last_undegraded":"2024-03-16T14:31:45.011021+0000","last_fullsized":"2024-03-16T14:31:45.011021+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:11:10.755811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114541+0000","last_change":"2024-03-16T14:31:08.710092+0000","last_active":"2024-03-16T14:31:45.114541+0000","last_peered":"2024-03-16T14:31:45.114541+0000","last_clean":"2024-03-16T14:31:45.114541+0000","last_became_active":"2024-03-16T14:31:08.709756+0000","last_became_peered":"2024-03-16T14:31:08.709756+0000","last_unstale":"2024-03-16T14:31:45.114541+0000","last_undegraded":"2024-03-16T14:31:45.114541+0000","last_fullsized":"2024-03-16T14:31:45.114541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:32:39.637196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268946+0000","last_change":"2024-03-16T14:31:11.202267+0000","last_active":"2024-03-16T14:31:45.268946+0000","last_peered":"2024-03-16T14:31:45.268946+0000","last_clean":"2024-03-16T14:31:45.268946+0000","last_became_active":"2024-03-16T14:31:11.201828+0000","last_became_peered":"2024-03-16T14:31:11.201828+0000","last_unstale":"2024-03-16T14:31:45.268946+0000","last_undegraded":"2024-03-16T14:31:45.268946+0000","last_fullsized":"2024-03-16T14:31:45.268946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:00:57.106478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269971+0000","last_change":"2024-03-16T14:31:04.676172+0000","last_active":"2024-03-16T14:31:45.269971+0000","last_peered":"2024-03-16T14:31:45.269971+0000","last_clean":"2024-03-16T14:31:45.269971+0000","last_became_active":"2024-03-16T14:31:04.675680+0000","last_became_peered":"2024-03-16T14:31:04.675680+0000","last_unstale":"2024-03-16T14:31:45.269971+0000","last_undegraded":"2024-03-16T14:31:45.269971+0000","last_fullsized":"2024-03-16T14:31:45.269971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:45.937214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011183+0000","last_change":"2024-03-16T14:31:06.697679+0000","last_active":"2024-03-16T14:31:45.011183+0000","last_peered":"2024-03-16T14:31:45.011183+0000","last_clean":"2024-03-16T14:31:45.011183+0000","last_became_active":"2024-03-16T14:31:06.697353+0000","last_became_peered":"2024-03-16T14:31:06.697353+0000","last_unstale":"2024-03-16T14:31:45.011183+0000","last_undegraded":"2024-03-16T14:31:45.011183+0000","last_fullsized":"2024-03-16T14:31:45.011183+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:49:06.836585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268797+0000","last_change":"2024-03-16T14:31:08.710342+0000","last_active":"2024-03-16T14:31:45.268797+0000","last_peered":"2024-03-16T14:31:45.268797+0000","last_clean":"2024-03-16T14:31:45.268797+0000","last_became_active":"2024-03-16T14:31:08.710005+0000","last_became_peered":"2024-03-16T14:31:08.710005+0000","last_unstale":"2024-03-16T14:31:45.268797+0000","last_undegraded":"2024-03-16T14:31:45.268797+0000","last_fullsized":"2024-03-16T14:31:45.268797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:49.614300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008350+0000","last_change":"2024-03-16T14:31:10.723297+0000","last_active":"2024-03-16T14:31:45.008350+0000","last_peered":"2024-03-16T14:31:45.008350+0000","last_clean":"2024-03-16T14:31:45.008350+0000","last_became_active":"2024-03-16T14:31:10.722665+0000","last_became_peered":"2024-03-16T14:31:10.722665+0000","last_unstale":"2024-03-16T14:31:45.008350+0000","last_undegraded":"2024-03-16T14:31:45.008350+0000","last_fullsized":"2024-03-16T14:31:45.008350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:47:25.111344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114257+0000","last_change":"2024-03-16T14:31:04.671435+0000","last_active":"2024-03-16T14:31:45.114257+0000","last_peered":"2024-03-16T14:31:45.114257+0000","last_clean":"2024-03-16T14:31:45.114257+0000","last_became_active":"2024-03-16T14:31:04.671132+0000","last_became_peered":"2024-03-16T14:31:04.671132+0000","last_unstale":"2024-03-16T14:31:45.114257+0000","last_undegraded":"2024-03-16T14:31:45.114257+0000","last_fullsized":"2024-03-16T14:31:45.114257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:05:53.732301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269509+0000","last_change":"2024-03-16T14:31:06.696481+0000","last_active":"2024-03-16T14:31:45.269509+0000","last_peered":"2024-03-16T14:31:45.269509+0000","last_clean":"2024-03-16T14:31:45.269509+0000","last_became_active":"2024-03-16T14:31:06.696124+0000","last_became_peered":"2024-03-16T14:31:06.696124+0000","last_unstale":"2024-03-16T14:31:45.269509+0000","last_undegraded":"2024-03-16T14:31:45.269509+0000","last_fullsized":"2024-03-16T14:31:45.269509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:07.025616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734552+0000","last_change":"2024-03-16T14:31:08.714259+0000","last_active":"2024-03-16T14:32:19.734552+0000","last_peered":"2024-03-16T14:32:19.734552+0000","last_clean":"2024-03-16T14:32:19.734552+0000","last_became_active":"2024-03-16T14:31:08.713926+0000","last_became_peered":"2024-03-16T14:31:08.713926+0000","last_unstale":"2024-03-16T14:32:19.734552+0000","last_undegraded":"2024-03-16T14:32:19.734552+0000","last_fullsized":"2024-03-16T14:32:19.734552+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:30:42.410143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129145+0000","last_change":"2024-03-16T14:31:11.204673+0000","last_active":"2024-03-16T14:31:45.129145+0000","last_peered":"2024-03-16T14:31:45.129145+0000","last_clean":"2024-03-16T14:31:45.129145+0000","last_became_active":"2024-03-16T14:31:11.204349+0000","last_became_peered":"2024-03-16T14:31:11.204349+0000","last_unstale":"2024-03-16T14:31:45.129145+0000","last_undegraded":"2024-03-16T14:31:45.129145+0000","last_fullsized":"2024-03-16T14:31:45.129145+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:56.361934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011154+0000","last_change":"2024-03-16T14:31:04.675031+0000","last_active":"2024-03-16T14:31:45.011154+0000","last_peered":"2024-03-16T14:31:45.011154+0000","last_clean":"2024-03-16T14:31:45.011154+0000","last_became_active":"2024-03-16T14:31:04.674382+0000","last_became_peered":"2024-03-16T14:31:04.674382+0000","last_unstale":"2024-03-16T14:31:45.011154+0000","last_undegraded":"2024-03-16T14:31:45.011154+0000","last_fullsized":"2024-03-16T14:31:45.011154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:16.718987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008219+0000","last_change":"2024-03-16T14:31:06.697569+0000","last_active":"2024-03-16T14:31:45.008219+0000","last_peered":"2024-03-16T14:31:45.008219+0000","last_clean":"2024-03-16T14:31:45.008219+0000","last_became_active":"2024-03-16T14:31:06.697220+0000","last_became_peered":"2024-03-16T14:31:06.697220+0000","last_unstale":"2024-03-16T14:31:45.008219+0000","last_undegraded":"2024-03-16T14:31:45.008219+0000","last_fullsized":"2024-03-16T14:31:45.008219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:42:25.829258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734509+0000","last_change":"2024-03-16T14:31:08.712851+0000","last_active":"2024-03-16T14:32:19.734509+0000","last_peered":"2024-03-16T14:32:19.734509+0000","last_clean":"2024-03-16T14:32:19.734509+0000","last_became_active":"2024-03-16T14:31:08.712514+0000","last_became_peered":"2024-03-16T14:31:08.712514+0000","last_unstale":"2024-03-16T14:32:19.734509+0000","last_undegraded":"2024-03-16T14:32:19.734509+0000","last_fullsized":"2024-03-16T14:32:19.734509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:47:34.072218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008389+0000","last_change":"2024-03-16T14:31:10.716362+0000","last_active":"2024-03-16T14:31:45.008389+0000","last_peered":"2024-03-16T14:31:45.008389+0000","last_clean":"2024-03-16T14:31:45.008389+0000","last_became_active":"2024-03-16T14:31:10.716038+0000","last_became_peered":"2024-03-16T14:31:10.716038+0000","last_unstale":"2024-03-16T14:31:45.008389+0000","last_undegraded":"2024-03-16T14:31:45.008389+0000","last_fullsized":"2024-03-16T14:31:45.008389+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:32.291747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270709+0000","last_change":"2024-03-16T14:31:04.670979+0000","last_active":"2024-03-16T14:31:45.270709+0000","last_peered":"2024-03-16T14:31:45.270709+0000","last_clean":"2024-03-16T14:31:45.270709+0000","last_became_active":"2024-03-16T14:31:04.670667+0000","last_became_peered":"2024-03-16T14:31:04.670667+0000","last_unstale":"2024-03-16T14:31:45.270709+0000","last_undegraded":"2024-03-16T14:31:45.270709+0000","last_fullsized":"2024-03-16T14:31:45.270709+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:15.354541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269073+0000","last_change":"2024-03-16T14:31:06.703641+0000","last_active":"2024-03-16T14:31:45.269073+0000","last_peered":"2024-03-16T14:31:45.269073+0000","last_clean":"2024-03-16T14:31:45.269073+0000","last_became_active":"2024-03-16T14:31:06.703234+0000","last_became_peered":"2024-03-16T14:31:06.703234+0000","last_unstale":"2024-03-16T14:31:45.269073+0000","last_undegraded":"2024-03-16T14:31:45.269073+0000","last_fullsized":"2024-03-16T14:31:45.269073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:00.265402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.007993+0000","last_change":"2024-03-16T14:31:08.712346+0000","last_active":"2024-03-16T14:31:45.007993+0000","last_peered":"2024-03-16T14:31:45.007993+0000","last_clean":"2024-03-16T14:31:45.007993+0000","last_became_active":"2024-03-16T14:31:08.711176+0000","last_became_peered":"2024-03-16T14:31:08.711176+0000","last_unstale":"2024-03-16T14:31:45.007993+0000","last_undegraded":"2024-03-16T14:31:45.007993+0000","last_fullsized":"2024-03-16T14:31:45.007993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:33.415351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114074+0000","last_change":"2024-03-16T14:31:10.719134+0000","last_active":"2024-03-16T14:31:45.114074+0000","last_peered":"2024-03-16T14:31:45.114074+0000","last_clean":"2024-03-16T14:31:45.114074+0000","last_became_active":"2024-03-16T14:31:10.718753+0000","last_became_peered":"2024-03-16T14:31:10.718753+0000","last_unstale":"2024-03-16T14:31:45.114074+0000","last_undegraded":"2024-03-16T14:31:45.114074+0000","last_fullsized":"2024-03-16T14:31:45.114074+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:53:58.407790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734561+0000","last_change":"2024-03-16T14:31:08.707754+0000","last_active":"2024-03-16T14:32:19.734561+0000","last_peered":"2024-03-16T14:32:19.734561+0000","last_clean":"2024-03-16T14:32:19.734561+0000","last_became_active":"2024-03-16T14:31:08.707408+0000","last_became_peered":"2024-03-16T14:31:08.707408+0000","last_unstale":"2024-03-16T14:32:19.734561+0000","last_undegraded":"2024-03-16T14:32:19.734561+0000","last_fullsized":"2024-03-16T14:32:19.734561+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:09.053922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268976+0000","last_change":"2024-03-16T14:31:04.677127+0000","last_active":"2024-03-16T14:31:45.268976+0000","last_peered":"2024-03-16T14:31:45.268976+0000","last_clean":"2024-03-16T14:31:45.268976+0000","last_became_active":"2024-03-16T14:31:04.676790+0000","last_became_peered":"2024-03-16T14:31:04.676790+0000","last_unstale":"2024-03-16T14:31:45.268976+0000","last_undegraded":"2024-03-16T14:31:45.268976+0000","last_fullsized":"2024-03-16T14:31:45.268976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:47.401589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270594+0000","last_change":"2024-03-16T14:31:06.699015+0000","last_active":"2024-03-16T14:31:45.270594+0000","last_peered":"2024-03-16T14:31:45.270594+0000","last_clean":"2024-03-16T14:31:45.270594+0000","last_became_active":"2024-03-16T14:31:06.698694+0000","last_became_peered":"2024-03-16T14:31:06.698694+0000","last_unstale":"2024-03-16T14:31:45.270594+0000","last_undegraded":"2024-03-16T14:31:45.270594+0000","last_fullsized":"2024-03-16T14:31:45.270594+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:11:36.443865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269452+0000","last_change":"2024-03-16T14:31:11.203288+0000","last_active":"2024-03-16T14:31:45.269452+0000","last_peered":"2024-03-16T14:31:45.269452+0000","last_clean":"2024-03-16T14:31:45.269452+0000","last_became_active":"2024-03-16T14:31:11.201970+0000","last_became_peered":"2024-03-16T14:31:11.201970+0000","last_unstale":"2024-03-16T14:31:45.269452+0000","last_undegraded":"2024-03-16T14:31:45.269452+0000","last_fullsized":"2024-03-16T14:31:45.269452+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:56:39.992885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114487+0000","last_change":"2024-03-16T14:31:08.711780+0000","last_active":"2024-03-16T14:31:45.114487+0000","last_peered":"2024-03-16T14:31:45.114487+0000","last_clean":"2024-03-16T14:31:45.114487+0000","last_became_active":"2024-03-16T14:31:08.710957+0000","last_became_peered":"2024-03-16T14:31:08.710957+0000","last_unstale":"2024-03-16T14:31:45.114487+0000","last_undegraded":"2024-03-16T14:31:45.114487+0000","last_fullsized":"2024-03-16T14:31:45.114487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:56.398499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008080+0000","last_change":"2024-03-16T14:31:04.672961+0000","last_active":"2024-03-16T14:31:45.008080+0000","last_peered":"2024-03-16T14:31:45.008080+0000","last_clean":"2024-03-16T14:31:45.008080+0000","last_became_active":"2024-03-16T14:31:04.672580+0000","last_became_peered":"2024-03-16T14:31:04.672580+0000","last_unstale":"2024-03-16T14:31:45.008080+0000","last_undegraded":"2024-03-16T14:31:45.008080+0000","last_fullsized":"2024-03-16T14:31:45.008080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:39:27.841097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010104+0000","last_change":"2024-03-16T14:31:06.702337+0000","last_active":"2024-03-16T14:31:45.010104+0000","last_peered":"2024-03-16T14:31:45.010104+0000","last_clean":"2024-03-16T14:31:45.010104+0000","last_became_active":"2024-03-16T14:31:06.701895+0000","last_became_peered":"2024-03-16T14:31:06.701895+0000","last_unstale":"2024-03-16T14:31:45.010104+0000","last_undegraded":"2024-03-16T14:31:45.010104+0000","last_fullsized":"2024-03-16T14:31:45.010104+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:37.149233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121878+0000","last_change":"2024-03-16T14:31:10.719020+0000","last_active":"2024-03-16T14:31:45.121878+0000","last_peered":"2024-03-16T14:31:45.121878+0000","last_clean":"2024-03-16T14:31:45.121878+0000","last_became_active":"2024-03-16T14:31:10.718666+0000","last_became_peered":"2024-03-16T14:31:10.718666+0000","last_unstale":"2024-03-16T14:31:45.121878+0000","last_undegraded":"2024-03-16T14:31:45.121878+0000","last_fullsized":"2024-03-16T14:31:45.121878+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:40.679127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270132+0000","last_change":"2024-03-16T14:31:08.716074+0000","last_active":"2024-03-16T14:31:45.270132+0000","last_peered":"2024-03-16T14:31:45.270132+0000","last_clean":"2024-03-16T14:31:45.270132+0000","last_became_active":"2024-03-16T14:31:08.715693+0000","last_became_peered":"2024-03-16T14:31:08.715693+0000","last_unstale":"2024-03-16T14:31:45.270132+0000","last_undegraded":"2024-03-16T14:31:45.270132+0000","last_fullsized":"2024-03-16T14:31:45.270132+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:07.054526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268413+0000","last_change":"2024-03-16T14:31:04.674235+0000","last_active":"2024-03-16T14:31:45.268413+0000","last_peered":"2024-03-16T14:31:45.268413+0000","last_clean":"2024-03-16T14:31:45.268413+0000","last_became_active":"2024-03-16T14:31:04.673944+0000","last_became_peered":"2024-03-16T14:31:04.673944+0000","last_unstale":"2024-03-16T14:31:45.268413+0000","last_undegraded":"2024-03-16T14:31:45.268413+0000","last_fullsized":"2024-03-16T14:31:45.268413+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:09.832597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009683+0000","last_change":"2024-03-16T14:31:32.543771+0000","last_active":"2024-03-16T14:31:45.009683+0000","last_peered":"2024-03-16T14:31:45.009683+0000","last_clean":"2024-03-16T14:31:45.009683+0000","last_became_active":"2024-03-16T14:31:32.528585+0000","last_became_peered":"2024-03-16T14:31:32.528585+0000","last_unstale":"2024-03-16T14:31:45.009683+0000","last_undegraded":"2024-03-16T14:31:45.009683+0000","last_fullsized":"2024-03-16T14:31:45.009683+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:43:12.134675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010353+0000","last_change":"2024-03-16T14:31:10.719838+0000","last_active":"2024-03-16T14:31:45.010353+0000","last_peered":"2024-03-16T14:31:45.010353+0000","last_clean":"2024-03-16T14:31:45.010353+0000","last_became_active":"2024-03-16T14:31:10.719337+0000","last_became_peered":"2024-03-16T14:31:10.719337+0000","last_unstale":"2024-03-16T14:31:45.010353+0000","last_undegraded":"2024-03-16T14:31:45.010353+0000","last_fullsized":"2024-03-16T14:31:45.010353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:49.411444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734515+0000","last_change":"2024-03-16T14:31:08.709343+0000","last_active":"2024-03-16T14:32:19.734515+0000","last_peered":"2024-03-16T14:32:19.734515+0000","last_clean":"2024-03-16T14:32:19.734515+0000","last_became_active":"2024-03-16T14:31:08.708152+0000","last_became_peered":"2024-03-16T14:31:08.708152+0000","last_unstale":"2024-03-16T14:32:19.734515+0000","last_undegraded":"2024-03-16T14:32:19.734515+0000","last_fullsized":"2024-03-16T14:32:19.734515+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:01:00.356461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270278+0000","last_change":"2024-03-16T14:31:04.667057+0000","last_active":"2024-03-16T14:31:45.270278+0000","last_peered":"2024-03-16T14:31:45.270278+0000","last_clean":"2024-03-16T14:31:45.270278+0000","last_became_active":"2024-03-16T14:31:04.666730+0000","last_became_peered":"2024-03-16T14:31:04.666730+0000","last_unstale":"2024-03-16T14:31:45.270278+0000","last_undegraded":"2024-03-16T14:31:45.270278+0000","last_fullsized":"2024-03-16T14:31:45.270278+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:05:47.841848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009999+0000","last_change":"2024-03-16T14:31:06.702973+0000","last_active":"2024-03-16T14:31:45.009999+0000","last_peered":"2024-03-16T14:31:45.009999+0000","last_clean":"2024-03-16T14:31:45.009999+0000","last_became_active":"2024-03-16T14:31:06.702659+0000","last_became_peered":"2024-03-16T14:31:06.702659+0000","last_unstale":"2024-03-16T14:31:45.009999+0000","last_undegraded":"2024-03-16T14:31:45.009999+0000","last_fullsized":"2024-03-16T14:31:45.009999+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:22.664741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32540,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708516,"statfs":{"total":95990841344,"available":95957520384,"internally_reserved":0,"allocated":1392640,"data_stored":923102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28364,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712692,"statfs":{"total":95990841344,"available":95961796608,"internally_reserved":0,"allocated":1380352,"data_stored":923397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713224,"statfs":{"total":95990841344,"available":95962341376,"internally_reserved":0,"allocated":831488,"data_stored":333560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712680,"statfs":{"total":95990841344,"available":95961784320,"internally_reserved":0,"allocated":1392640,"data_stored":922831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709108,"statfs":{"total":95990841344,"available":95958126592,"internally_reserved":0,"allocated":786432,"data_stored":332453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713244,"statfs":{"total":95990841344,"available":95962361856,"internally_reserved":0,"allocated":811008,"data_stored":332560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709088,"statfs":{"total":95990841344,"available":95958106112,"internally_reserved":0,"allocated":806912,"data_stored":333185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713268,"statfs":{"total":95990841344,"available":95962386432,"internally_reserved":0,"allocated":786432,"data_stored":333016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T14:32:28.463 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-16T14:32:28.718 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:28.755 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:32:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:32:28.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:28.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:28.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T14:32:28.757 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:28.758 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-16T14:32:28.759 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:28 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-16T14:32:28.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:28 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:29.755 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:29 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:29.513Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00060433s 2024-03-16T14:32:29.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:29 smithi026 ceph-mon[25218]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-16T14:32:29.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:29 smithi026 ceph-mon[25218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-16T14:32:29.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:29 smithi026 ceph-mon[25218]: Deploying daemon grafana.a on smithi151 2024-03-16T14:32:29.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:29 smithi026 ceph-mon[38287]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-16T14:32:29.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:29 smithi026 ceph-mon[38287]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-16T14:32:29.756 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:29 smithi026 ceph-mon[38287]: Deploying daemon grafana.a on smithi151 2024-03-16T14:32:29.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:29 smithi151 ceph-mon[30603]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-16T14:32:29.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:29 smithi151 ceph-mon[30603]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-16T14:32:29.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:29 smithi151 ceph-mon[30603]: Deploying daemon grafana.a on smithi151 2024-03-16T14:32:30.637 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:32:30.639 INFO:teuthology.orchestra.run.smithi026.stderr:dumped all 2024-03-16T14:32:30.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:30 smithi151 ceph-mon[30603]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:30 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:30.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:30 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:30 smithi026 ceph-mon[25218]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:30 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:31.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:30 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:31.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:30 smithi026 ceph-mon[38287]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:31.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:30 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:31.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:30 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:31.440 INFO:teuthology.orchestra.run.smithi026.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-16T14:32:29.030201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1039,"num_read_kb":882,"num_write":681,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":660,"ondisk_log_size":660,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":236628,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749691820,"statfs":{"total":767926730752,"available":767684423680,"internally_reserved":0,"allocated":8187904,"data_stored":4434104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003139"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008286+0000","last_change":"2024-03-16T14:31:10.719085+0000","last_active":"2024-03-16T14:31:45.008286+0000","last_peered":"2024-03-16T14:31:45.008286+0000","last_clean":"2024-03-16T14:31:45.008286+0000","last_became_active":"2024-03-16T14:31:10.718694+0000","last_became_peered":"2024-03-16T14:31:10.718694+0000","last_unstale":"2024-03-16T14:31:45.008286+0000","last_undegraded":"2024-03-16T14:31:45.008286+0000","last_fullsized":"2024-03-16T14:31:45.008286+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:10:47.412379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269048+0000","last_change":"2024-03-16T14:31:08.715961+0000","last_active":"2024-03-16T14:31:45.269048+0000","last_peered":"2024-03-16T14:31:45.269048+0000","last_clean":"2024-03-16T14:31:45.269048+0000","last_became_active":"2024-03-16T14:31:08.715607+0000","last_became_peered":"2024-03-16T14:31:08.715607+0000","last_unstale":"2024-03-16T14:31:45.269048+0000","last_undegraded":"2024-03-16T14:31:45.269048+0000","last_fullsized":"2024-03-16T14:31:45.269048+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:04:11.485717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010516+0000","last_change":"2024-03-16T14:31:04.671856+0000","last_active":"2024-03-16T14:31:45.010516+0000","last_peered":"2024-03-16T14:31:45.010516+0000","last_clean":"2024-03-16T14:31:45.010516+0000","last_became_active":"2024-03-16T14:31:04.671247+0000","last_became_peered":"2024-03-16T14:31:04.671247+0000","last_unstale":"2024-03-16T14:31:45.010516+0000","last_undegraded":"2024-03-16T14:31:45.010516+0000","last_fullsized":"2024-03-16T14:31:45.010516+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:07:17.068232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.007949+0000","last_change":"2024-03-16T14:31:06.697380+0000","last_active":"2024-03-16T14:31:45.007949+0000","last_peered":"2024-03-16T14:31:45.007949+0000","last_clean":"2024-03-16T14:31:45.007949+0000","last_became_active":"2024-03-16T14:31:06.697044+0000","last_became_peered":"2024-03-16T14:31:06.697044+0000","last_unstale":"2024-03-16T14:31:45.007949+0000","last_undegraded":"2024-03-16T14:31:45.007949+0000","last_fullsized":"2024-03-16T14:31:45.007949+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:37:22.253099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270099+0000","last_change":"2024-03-16T14:31:11.206078+0000","last_active":"2024-03-16T14:31:45.270099+0000","last_peered":"2024-03-16T14:31:45.270099+0000","last_clean":"2024-03-16T14:31:45.270099+0000","last_became_active":"2024-03-16T14:31:11.205524+0000","last_became_peered":"2024-03-16T14:31:11.205524+0000","last_unstale":"2024-03-16T14:31:45.270099+0000","last_undegraded":"2024-03-16T14:31:45.270099+0000","last_fullsized":"2024-03-16T14:31:45.270099+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:24:57.860915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.007952+0000","last_change":"2024-03-16T14:31:08.706796+0000","last_active":"2024-03-16T14:31:45.007952+0000","last_peered":"2024-03-16T14:31:45.007952+0000","last_clean":"2024-03-16T14:31:45.007952+0000","last_became_active":"2024-03-16T14:31:08.706445+0000","last_became_peered":"2024-03-16T14:31:08.706445+0000","last_unstale":"2024-03-16T14:31:45.007952+0000","last_undegraded":"2024-03-16T14:31:45.007952+0000","last_fullsized":"2024-03-16T14:31:45.007952+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:44:48.515244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270635+0000","last_change":"2024-03-16T14:31:04.669846+0000","last_active":"2024-03-16T14:31:45.270635+0000","last_peered":"2024-03-16T14:31:45.270635+0000","last_clean":"2024-03-16T14:31:45.270635+0000","last_became_active":"2024-03-16T14:31:04.669538+0000","last_became_peered":"2024-03-16T14:31:04.669538+0000","last_unstale":"2024-03-16T14:31:45.270635+0000","last_undegraded":"2024-03-16T14:31:45.270635+0000","last_fullsized":"2024-03-16T14:31:45.270635+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:08:21.155953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269692+0000","last_change":"2024-03-16T14:31:06.701679+0000","last_active":"2024-03-16T14:31:45.269692+0000","last_peered":"2024-03-16T14:31:45.269692+0000","last_clean":"2024-03-16T14:31:45.269692+0000","last_became_active":"2024-03-16T14:31:06.701250+0000","last_became_peered":"2024-03-16T14:31:06.701250+0000","last_unstale":"2024-03-16T14:31:45.269692+0000","last_undegraded":"2024-03-16T14:31:45.269692+0000","last_fullsized":"2024-03-16T14:31:45.269692+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:58:50.977684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008545+0000","last_change":"2024-03-16T14:31:10.720081+0000","last_active":"2024-03-16T14:31:45.008545+0000","last_peered":"2024-03-16T14:31:45.008545+0000","last_clean":"2024-03-16T14:31:45.008545+0000","last_became_active":"2024-03-16T14:31:10.719496+0000","last_became_peered":"2024-03-16T14:31:10.719496+0000","last_unstale":"2024-03-16T14:31:45.008545+0000","last_undegraded":"2024-03-16T14:31:45.008545+0000","last_fullsized":"2024-03-16T14:31:45.008545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:03:39.356092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113846+0000","last_change":"2024-03-16T14:31:08.708031+0000","last_active":"2024-03-16T14:31:45.113846+0000","last_peered":"2024-03-16T14:31:45.113846+0000","last_clean":"2024-03-16T14:31:45.113846+0000","last_became_active":"2024-03-16T14:31:08.707031+0000","last_became_peered":"2024-03-16T14:31:08.707031+0000","last_unstale":"2024-03-16T14:31:45.113846+0000","last_undegraded":"2024-03-16T14:31:45.113846+0000","last_fullsized":"2024-03-16T14:31:45.113846+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:49:11.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114025+0000","last_change":"2024-03-16T14:31:04.670431+0000","last_active":"2024-03-16T14:31:45.114025+0000","last_peered":"2024-03-16T14:31:45.114025+0000","last_clean":"2024-03-16T14:31:45.114025+0000","last_became_active":"2024-03-16T14:31:04.670134+0000","last_became_peered":"2024-03-16T14:31:04.670134+0000","last_unstale":"2024-03-16T14:31:45.114025+0000","last_undegraded":"2024-03-16T14:31:45.114025+0000","last_fullsized":"2024-03-16T14:31:45.114025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:14:22.548560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008542+0000","last_change":"2024-03-16T14:31:06.703942+0000","last_active":"2024-03-16T14:31:45.008542+0000","last_peered":"2024-03-16T14:31:45.008542+0000","last_clean":"2024-03-16T14:31:45.008542+0000","last_became_active":"2024-03-16T14:31:06.703579+0000","last_became_peered":"2024-03-16T14:31:06.703579+0000","last_unstale":"2024-03-16T14:31:45.008542+0000","last_undegraded":"2024-03-16T14:31:45.008542+0000","last_fullsized":"2024-03-16T14:31:45.008542+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:58:51.973773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113685+0000","last_change":"2024-03-16T14:31:10.726539+0000","last_active":"2024-03-16T14:31:45.113685+0000","last_peered":"2024-03-16T14:31:45.113685+0000","last_clean":"2024-03-16T14:31:45.113685+0000","last_became_active":"2024-03-16T14:31:10.726241+0000","last_became_peered":"2024-03-16T14:31:10.726241+0000","last_unstale":"2024-03-16T14:31:45.113685+0000","last_undegraded":"2024-03-16T14:31:45.113685+0000","last_fullsized":"2024-03-16T14:31:45.113685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:21:12.298340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128686+0000","last_change":"2024-03-16T14:31:08.708166+0000","last_active":"2024-03-16T14:31:45.128686+0000","last_peered":"2024-03-16T14:31:45.128686+0000","last_clean":"2024-03-16T14:31:45.128686+0000","last_became_active":"2024-03-16T14:31:08.707798+0000","last_became_peered":"2024-03-16T14:31:08.707798+0000","last_unstale":"2024-03-16T14:31:45.128686+0000","last_undegraded":"2024-03-16T14:31:45.128686+0000","last_fullsized":"2024-03-16T14:31:45.128686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:03:51.224667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114019+0000","last_change":"2024-03-16T14:31:04.674726+0000","last_active":"2024-03-16T14:31:45.114019+0000","last_peered":"2024-03-16T14:31:45.114019+0000","last_clean":"2024-03-16T14:31:45.114019+0000","last_became_active":"2024-03-16T14:31:04.674469+0000","last_became_peered":"2024-03-16T14:31:04.674469+0000","last_unstale":"2024-03-16T14:31:45.114019+0000","last_undegraded":"2024-03-16T14:31:45.114019+0000","last_fullsized":"2024-03-16T14:31:45.114019+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:48:15.554153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'5","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008949+0000","last_change":"2024-03-16T14:31:06.703886+0000","last_active":"2024-03-16T14:31:45.008949+0000","last_peered":"2024-03-16T14:31:45.008949+0000","last_clean":"2024-03-16T14:31:45.008949+0000","last_became_active":"2024-03-16T14:31:06.703513+0000","last_became_peered":"2024-03-16T14:31:06.703513+0000","last_unstale":"2024-03-16T14:31:45.008949+0000","last_undegraded":"2024-03-16T14:31:45.008949+0000","last_fullsized":"2024-03-16T14:31:45.008949+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:23:54.108322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010254+0000","last_change":"2024-03-16T14:31:11.202040+0000","last_active":"2024-03-16T14:31:45.010254+0000","last_peered":"2024-03-16T14:31:45.010254+0000","last_clean":"2024-03-16T14:31:45.010254+0000","last_became_active":"2024-03-16T14:31:11.201729+0000","last_became_peered":"2024-03-16T14:31:11.201729+0000","last_unstale":"2024-03-16T14:31:45.010254+0000","last_undegraded":"2024-03-16T14:31:45.010254+0000","last_fullsized":"2024-03-16T14:31:45.010254+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:22:44.615838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270280+0000","last_change":"2024-03-16T14:31:08.716241+0000","last_active":"2024-03-16T14:31:45.270280+0000","last_peered":"2024-03-16T14:31:45.270280+0000","last_clean":"2024-03-16T14:31:45.270280+0000","last_became_active":"2024-03-16T14:31:08.715713+0000","last_became_peered":"2024-03-16T14:31:08.715713+0000","last_unstale":"2024-03-16T14:31:45.270280+0000","last_undegraded":"2024-03-16T14:31:45.270280+0000","last_fullsized":"2024-03-16T14:31:45.270280+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:07:18.500434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011099+0000","last_change":"2024-03-16T14:31:04.677477+0000","last_active":"2024-03-16T14:31:45.011099+0000","last_peered":"2024-03-16T14:31:45.011099+0000","last_clean":"2024-03-16T14:31:45.011099+0000","last_became_active":"2024-03-16T14:31:04.677169+0000","last_became_peered":"2024-03-16T14:31:04.677169+0000","last_unstale":"2024-03-16T14:31:45.011099+0000","last_undegraded":"2024-03-16T14:31:45.011099+0000","last_fullsized":"2024-03-16T14:31:45.011099+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:53:14.367957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121930+0000","last_change":"2024-03-16T14:31:06.694617+0000","last_active":"2024-03-16T14:31:45.121930+0000","last_peered":"2024-03-16T14:31:45.121930+0000","last_clean":"2024-03-16T14:31:45.121930+0000","last_became_active":"2024-03-16T14:31:06.694260+0000","last_became_peered":"2024-03-16T14:31:06.694260+0000","last_unstale":"2024-03-16T14:31:45.121930+0000","last_undegraded":"2024-03-16T14:31:45.121930+0000","last_fullsized":"2024-03-16T14:31:45.121930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:36:19.724889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270937+0000","last_change":"2024-03-16T14:31:11.202242+0000","last_active":"2024-03-16T14:31:45.270937+0000","last_peered":"2024-03-16T14:31:45.270937+0000","last_clean":"2024-03-16T14:31:45.270937+0000","last_became_active":"2024-03-16T14:31:11.201849+0000","last_became_peered":"2024-03-16T14:31:11.201849+0000","last_unstale":"2024-03-16T14:31:45.270937+0000","last_undegraded":"2024-03-16T14:31:45.270937+0000","last_fullsized":"2024-03-16T14:31:45.270937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:33:04.048437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009906+0000","last_change":"2024-03-16T14:31:08.708199+0000","last_active":"2024-03-16T14:31:45.009906+0000","last_peered":"2024-03-16T14:31:45.009906+0000","last_clean":"2024-03-16T14:31:45.009906+0000","last_became_active":"2024-03-16T14:31:08.707881+0000","last_became_peered":"2024-03-16T14:31:08.707881+0000","last_unstale":"2024-03-16T14:31:45.009906+0000","last_undegraded":"2024-03-16T14:31:45.009906+0000","last_fullsized":"2024-03-16T14:31:45.009906+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:45:42.193528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008670+0000","last_change":"2024-03-16T14:31:04.674838+0000","last_active":"2024-03-16T14:31:45.008670+0000","last_peered":"2024-03-16T14:31:45.008670+0000","last_clean":"2024-03-16T14:31:45.008670+0000","last_became_active":"2024-03-16T14:31:04.674538+0000","last_became_peered":"2024-03-16T14:31:04.674538+0000","last_unstale":"2024-03-16T14:31:45.008670+0000","last_undegraded":"2024-03-16T14:31:45.008670+0000","last_fullsized":"2024-03-16T14:31:45.008670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:30:09.972153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'12","reported_seq":70,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270208+0000","last_change":"2024-03-16T14:31:31.638092+0000","last_active":"2024-03-16T14:31:45.270208+0000","last_peered":"2024-03-16T14:31:45.270208+0000","last_clean":"2024-03-16T14:31:45.270208+0000","last_became_active":"2024-03-16T14:31:31.631485+0000","last_became_peered":"2024-03-16T14:31:31.631485+0000","last_unstale":"2024-03-16T14:31:45.270208+0000","last_undegraded":"2024-03-16T14:31:45.270208+0000","last_fullsized":"2024-03-16T14:31:45.270208+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:26:46.750504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128904+0000","last_change":"2024-03-16T14:31:10.721874+0000","last_active":"2024-03-16T14:31:45.128904+0000","last_peered":"2024-03-16T14:31:45.128904+0000","last_clean":"2024-03-16T14:31:45.128904+0000","last_became_active":"2024-03-16T14:31:10.721564+0000","last_became_peered":"2024-03-16T14:31:10.721564+0000","last_unstale":"2024-03-16T14:31:45.128904+0000","last_undegraded":"2024-03-16T14:31:45.128904+0000","last_fullsized":"2024-03-16T14:31:45.128904+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:17:36.459882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734558+0000","last_change":"2024-03-16T14:31:08.714781+0000","last_active":"2024-03-16T14:32:19.734558+0000","last_peered":"2024-03-16T14:32:19.734558+0000","last_clean":"2024-03-16T14:32:19.734558+0000","last_became_active":"2024-03-16T14:31:08.712843+0000","last_became_peered":"2024-03-16T14:31:08.712843+0000","last_unstale":"2024-03-16T14:32:19.734558+0000","last_undegraded":"2024-03-16T14:32:19.734558+0000","last_fullsized":"2024-03-16T14:32:19.734558+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:46:04.108601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269150+0000","last_change":"2024-03-16T14:31:04.678511+0000","last_active":"2024-03-16T14:31:45.269150+0000","last_peered":"2024-03-16T14:31:45.269150+0000","last_clean":"2024-03-16T14:31:45.269150+0000","last_became_active":"2024-03-16T14:31:04.678092+0000","last_became_peered":"2024-03-16T14:31:04.678092+0000","last_unstale":"2024-03-16T14:31:45.269150+0000","last_undegraded":"2024-03-16T14:31:45.269150+0000","last_fullsized":"2024-03-16T14:31:45.269150+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:47:12.439794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011259+0000","last_change":"2024-03-16T14:31:06.693954+0000","last_active":"2024-03-16T14:31:45.011259+0000","last_peered":"2024-03-16T14:31:45.011259+0000","last_clean":"2024-03-16T14:31:45.011259+0000","last_became_active":"2024-03-16T14:31:06.693583+0000","last_became_peered":"2024-03-16T14:31:06.693583+0000","last_unstale":"2024-03-16T14:31:45.011259+0000","last_undegraded":"2024-03-16T14:31:45.011259+0000","last_fullsized":"2024-03-16T14:31:45.011259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:02:07.726396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270379+0000","last_change":"2024-03-16T14:31:10.720149+0000","last_active":"2024-03-16T14:31:45.270379+0000","last_peered":"2024-03-16T14:31:45.270379+0000","last_clean":"2024-03-16T14:31:45.270379+0000","last_became_active":"2024-03-16T14:31:10.719813+0000","last_became_peered":"2024-03-16T14:31:10.719813+0000","last_unstale":"2024-03-16T14:31:45.270379+0000","last_undegraded":"2024-03-16T14:31:45.270379+0000","last_fullsized":"2024-03-16T14:31:45.270379+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:29:22.913842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011559+0000","last_change":"2024-03-16T14:31:08.709707+0000","last_active":"2024-03-16T14:31:45.011559+0000","last_peered":"2024-03-16T14:31:45.011559+0000","last_clean":"2024-03-16T14:31:45.011559+0000","last_became_active":"2024-03-16T14:31:08.709325+0000","last_became_peered":"2024-03-16T14:31:08.709325+0000","last_unstale":"2024-03-16T14:31:45.011559+0000","last_undegraded":"2024-03-16T14:31:45.011559+0000","last_fullsized":"2024-03-16T14:31:45.011559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:14:17.351327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268725+0000","last_change":"2024-03-16T14:31:04.677294+0000","last_active":"2024-03-16T14:31:45.268725+0000","last_peered":"2024-03-16T14:31:45.268725+0000","last_clean":"2024-03-16T14:31:45.268725+0000","last_became_active":"2024-03-16T14:31:04.676996+0000","last_became_peered":"2024-03-16T14:31:04.676996+0000","last_unstale":"2024-03-16T14:31:45.268725+0000","last_undegraded":"2024-03-16T14:31:45.268725+0000","last_fullsized":"2024-03-16T14:31:45.268725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:19:46.999303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268895+0000","last_change":"2024-03-16T14:31:06.704882+0000","last_active":"2024-03-16T14:31:45.268895+0000","last_peered":"2024-03-16T14:31:45.268895+0000","last_clean":"2024-03-16T14:31:45.268895+0000","last_became_active":"2024-03-16T14:31:06.704497+0000","last_became_peered":"2024-03-16T14:31:06.704497+0000","last_unstale":"2024-03-16T14:31:45.268895+0000","last_undegraded":"2024-03-16T14:31:45.268895+0000","last_fullsized":"2024-03-16T14:31:45.268895+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:04:01.825949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128632+0000","last_change":"2024-03-16T14:31:10.722335+0000","last_active":"2024-03-16T14:31:45.128632+0000","last_peered":"2024-03-16T14:31:45.128632+0000","last_clean":"2024-03-16T14:31:45.128632+0000","last_became_active":"2024-03-16T14:31:10.721682+0000","last_became_peered":"2024-03-16T14:31:10.721682+0000","last_unstale":"2024-03-16T14:31:45.128632+0000","last_undegraded":"2024-03-16T14:31:45.128632+0000","last_fullsized":"2024-03-16T14:31:45.128632+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:31:06.813691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268325+0000","last_change":"2024-03-16T14:31:08.714189+0000","last_active":"2024-03-16T14:31:45.268325+0000","last_peered":"2024-03-16T14:31:45.268325+0000","last_clean":"2024-03-16T14:31:45.268325+0000","last_became_active":"2024-03-16T14:31:08.713878+0000","last_became_peered":"2024-03-16T14:31:08.713878+0000","last_unstale":"2024-03-16T14:31:45.268325+0000","last_undegraded":"2024-03-16T14:31:45.268325+0000","last_fullsized":"2024-03-16T14:31:45.268325+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:28:06.729052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008775+0000","last_change":"2024-03-16T14:31:04.671108+0000","last_active":"2024-03-16T14:31:45.008775+0000","last_peered":"2024-03-16T14:31:45.008775+0000","last_clean":"2024-03-16T14:31:45.008775+0000","last_became_active":"2024-03-16T14:31:04.670757+0000","last_became_peered":"2024-03-16T14:31:04.670757+0000","last_unstale":"2024-03-16T14:31:45.008775+0000","last_undegraded":"2024-03-16T14:31:45.008775+0000","last_fullsized":"2024-03-16T14:31:45.008775+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:49:15.942210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268859+0000","last_change":"2024-03-16T14:31:06.699333+0000","last_active":"2024-03-16T14:31:45.268859+0000","last_peered":"2024-03-16T14:31:45.268859+0000","last_clean":"2024-03-16T14:31:45.268859+0000","last_became_active":"2024-03-16T14:31:06.698971+0000","last_became_peered":"2024-03-16T14:31:06.698971+0000","last_unstale":"2024-03-16T14:31:45.268859+0000","last_undegraded":"2024-03-16T14:31:45.268859+0000","last_fullsized":"2024-03-16T14:31:45.268859+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:05:17.236078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269917+0000","last_change":"2024-03-16T14:31:11.205614+0000","last_active":"2024-03-16T14:31:45.269917+0000","last_peered":"2024-03-16T14:31:45.269917+0000","last_clean":"2024-03-16T14:31:45.269917+0000","last_became_active":"2024-03-16T14:31:11.205274+0000","last_became_peered":"2024-03-16T14:31:11.205274+0000","last_unstale":"2024-03-16T14:31:45.269917+0000","last_undegraded":"2024-03-16T14:31:45.269917+0000","last_fullsized":"2024-03-16T14:31:45.269917+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:56:11.173148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128829+0000","last_change":"2024-03-16T14:31:08.708811+0000","last_active":"2024-03-16T14:31:45.128829+0000","last_peered":"2024-03-16T14:31:45.128829+0000","last_clean":"2024-03-16T14:31:45.128829+0000","last_became_active":"2024-03-16T14:31:08.708487+0000","last_became_peered":"2024-03-16T14:31:08.708487+0000","last_unstale":"2024-03-16T14:31:45.128829+0000","last_undegraded":"2024-03-16T14:31:45.128829+0000","last_fullsized":"2024-03-16T14:31:45.128829+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:58:10.235815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114106+0000","last_change":"2024-03-16T14:31:04.674161+0000","last_active":"2024-03-16T14:31:45.114106+0000","last_peered":"2024-03-16T14:31:45.114106+0000","last_clean":"2024-03-16T14:31:45.114106+0000","last_became_active":"2024-03-16T14:31:04.673858+0000","last_became_peered":"2024-03-16T14:31:04.673858+0000","last_unstale":"2024-03-16T14:31:45.114106+0000","last_undegraded":"2024-03-16T14:31:45.114106+0000","last_fullsized":"2024-03-16T14:31:45.114106+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:49:02.685654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269569+0000","last_change":"2024-03-16T14:31:06.698174+0000","last_active":"2024-03-16T14:31:45.269569+0000","last_peered":"2024-03-16T14:31:45.269569+0000","last_clean":"2024-03-16T14:31:45.269569+0000","last_became_active":"2024-03-16T14:31:06.697855+0000","last_became_peered":"2024-03-16T14:31:06.697855+0000","last_unstale":"2024-03-16T14:31:45.269569+0000","last_undegraded":"2024-03-16T14:31:45.269569+0000","last_fullsized":"2024-03-16T14:31:45.269569+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:03.834268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008216+0000","last_change":"2024-03-16T14:31:10.719826+0000","last_active":"2024-03-16T14:31:45.008216+0000","last_peered":"2024-03-16T14:31:45.008216+0000","last_clean":"2024-03-16T14:31:45.008216+0000","last_became_active":"2024-03-16T14:31:10.719059+0000","last_became_peered":"2024-03-16T14:31:10.719059+0000","last_unstale":"2024-03-16T14:31:45.008216+0000","last_undegraded":"2024-03-16T14:31:45.008216+0000","last_fullsized":"2024-03-16T14:31:45.008216+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:02:32.526482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270192+0000","last_change":"2024-03-16T14:31:08.707185+0000","last_active":"2024-03-16T14:31:45.270192+0000","last_peered":"2024-03-16T14:31:45.270192+0000","last_clean":"2024-03-16T14:31:45.270192+0000","last_became_active":"2024-03-16T14:31:08.706857+0000","last_became_peered":"2024-03-16T14:31:08.706857+0000","last_unstale":"2024-03-16T14:31:45.270192+0000","last_undegraded":"2024-03-16T14:31:45.270192+0000","last_fullsized":"2024-03-16T14:31:45.270192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:51:18.427073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128915+0000","last_change":"2024-03-16T14:31:04.677626+0000","last_active":"2024-03-16T14:31:45.128915+0000","last_peered":"2024-03-16T14:31:45.128915+0000","last_clean":"2024-03-16T14:31:45.128915+0000","last_became_active":"2024-03-16T14:31:04.677326+0000","last_became_peered":"2024-03-16T14:31:04.677326+0000","last_unstale":"2024-03-16T14:31:45.128915+0000","last_undegraded":"2024-03-16T14:31:45.128915+0000","last_fullsized":"2024-03-16T14:31:45.128915+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:08:14.371875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269655+0000","last_change":"2024-03-16T14:31:06.698345+0000","last_active":"2024-03-16T14:31:45.269655+0000","last_peered":"2024-03-16T14:31:45.269655+0000","last_clean":"2024-03-16T14:31:45.269655+0000","last_became_active":"2024-03-16T14:31:06.697407+0000","last_became_peered":"2024-03-16T14:31:06.697407+0000","last_unstale":"2024-03-16T14:31:45.269655+0000","last_undegraded":"2024-03-16T14:31:45.269655+0000","last_fullsized":"2024-03-16T14:31:45.269655+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:20:24.994698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268976+0000","last_change":"2024-03-16T14:31:10.722774+0000","last_active":"2024-03-16T14:31:45.268976+0000","last_peered":"2024-03-16T14:31:45.268976+0000","last_clean":"2024-03-16T14:31:45.268976+0000","last_became_active":"2024-03-16T14:31:10.722401+0000","last_became_peered":"2024-03-16T14:31:10.722401+0000","last_unstale":"2024-03-16T14:31:45.268976+0000","last_undegraded":"2024-03-16T14:31:45.268976+0000","last_fullsized":"2024-03-16T14:31:45.268976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:38:55.215778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269416+0000","last_change":"2024-03-16T14:31:08.714961+0000","last_active":"2024-03-16T14:31:45.269416+0000","last_peered":"2024-03-16T14:31:45.269416+0000","last_clean":"2024-03-16T14:31:45.269416+0000","last_became_active":"2024-03-16T14:31:08.713755+0000","last_became_peered":"2024-03-16T14:31:08.713755+0000","last_unstale":"2024-03-16T14:31:45.269416+0000","last_undegraded":"2024-03-16T14:31:45.269416+0000","last_fullsized":"2024-03-16T14:31:45.269416+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:47:31.213900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010185+0000","last_change":"2024-03-16T14:31:04.673573+0000","last_active":"2024-03-16T14:31:45.010185+0000","last_peered":"2024-03-16T14:31:45.010185+0000","last_clean":"2024-03-16T14:31:45.010185+0000","last_became_active":"2024-03-16T14:31:04.673201+0000","last_became_peered":"2024-03-16T14:31:04.673201+0000","last_unstale":"2024-03-16T14:31:45.010185+0000","last_undegraded":"2024-03-16T14:31:45.010185+0000","last_fullsized":"2024-03-16T14:31:45.010185+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:22:08.408440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009681+0000","last_change":"2024-03-16T14:31:06.700955+0000","last_active":"2024-03-16T14:31:45.009681+0000","last_peered":"2024-03-16T14:31:45.009681+0000","last_clean":"2024-03-16T14:31:45.009681+0000","last_became_active":"2024-03-16T14:31:06.700681+0000","last_became_peered":"2024-03-16T14:31:06.700681+0000","last_unstale":"2024-03-16T14:31:45.009681+0000","last_undegraded":"2024-03-16T14:31:45.009681+0000","last_fullsized":"2024-03-16T14:31:45.009681+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:17:34.317137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010919+0000","last_change":"2024-03-16T14:31:10.726683+0000","last_active":"2024-03-16T14:31:45.010919+0000","last_peered":"2024-03-16T14:31:45.010919+0000","last_clean":"2024-03-16T14:31:45.010919+0000","last_became_active":"2024-03-16T14:31:10.726274+0000","last_became_peered":"2024-03-16T14:31:10.726274+0000","last_unstale":"2024-03-16T14:31:45.010919+0000","last_undegraded":"2024-03-16T14:31:45.010919+0000","last_fullsized":"2024-03-16T14:31:45.010919+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:24:43.244002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734560+0000","last_change":"2024-03-16T14:31:08.706557+0000","last_active":"2024-03-16T14:32:19.734560+0000","last_peered":"2024-03-16T14:32:19.734560+0000","last_clean":"2024-03-16T14:32:19.734560+0000","last_became_active":"2024-03-16T14:31:08.706200+0000","last_became_peered":"2024-03-16T14:31:08.706200+0000","last_unstale":"2024-03-16T14:32:19.734560+0000","last_undegraded":"2024-03-16T14:32:19.734560+0000","last_fullsized":"2024-03-16T14:32:19.734560+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:38:43.435022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128759+0000","last_change":"2024-03-16T14:31:04.675942+0000","last_active":"2024-03-16T14:31:45.128759+0000","last_peered":"2024-03-16T14:31:45.128759+0000","last_clean":"2024-03-16T14:31:45.128759+0000","last_became_active":"2024-03-16T14:31:04.675663+0000","last_became_peered":"2024-03-16T14:31:04.675663+0000","last_unstale":"2024-03-16T14:31:45.128759+0000","last_undegraded":"2024-03-16T14:31:45.128759+0000","last_fullsized":"2024-03-16T14:31:45.128759+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:02.772572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269546+0000","last_change":"2024-03-16T14:31:06.691744+0000","last_active":"2024-03-16T14:31:45.269546+0000","last_peered":"2024-03-16T14:31:45.269546+0000","last_clean":"2024-03-16T14:31:45.269546+0000","last_became_active":"2024-03-16T14:31:06.691382+0000","last_became_peered":"2024-03-16T14:31:06.691382+0000","last_unstale":"2024-03-16T14:31:45.269546+0000","last_undegraded":"2024-03-16T14:31:45.269546+0000","last_fullsized":"2024-03-16T14:31:45.269546+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:51:06.922360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010661+0000","last_change":"2024-03-16T14:31:10.718741+0000","last_active":"2024-03-16T14:31:45.010661+0000","last_peered":"2024-03-16T14:31:45.010661+0000","last_clean":"2024-03-16T14:31:45.010661+0000","last_became_active":"2024-03-16T14:31:10.718431+0000","last_became_peered":"2024-03-16T14:31:10.718431+0000","last_unstale":"2024-03-16T14:31:45.010661+0000","last_undegraded":"2024-03-16T14:31:45.010661+0000","last_fullsized":"2024-03-16T14:31:45.010661+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:31:52.312816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734793+0000","last_change":"2024-03-16T14:31:08.705967+0000","last_active":"2024-03-16T14:32:19.734793+0000","last_peered":"2024-03-16T14:32:19.734793+0000","last_clean":"2024-03-16T14:32:19.734793+0000","last_became_active":"2024-03-16T14:31:08.705648+0000","last_became_peered":"2024-03-16T14:31:08.705648+0000","last_unstale":"2024-03-16T14:32:19.734793+0000","last_undegraded":"2024-03-16T14:32:19.734793+0000","last_fullsized":"2024-03-16T14:32:19.734793+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:37:59.643938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010830+0000","last_change":"2024-03-16T14:31:04.674328+0000","last_active":"2024-03-16T14:31:45.010830+0000","last_peered":"2024-03-16T14:31:45.010830+0000","last_clean":"2024-03-16T14:31:45.010830+0000","last_became_active":"2024-03-16T14:31:04.673998+0000","last_became_peered":"2024-03-16T14:31:04.673998+0000","last_unstale":"2024-03-16T14:31:45.010830+0000","last_undegraded":"2024-03-16T14:31:45.010830+0000","last_fullsized":"2024-03-16T14:31:45.010830+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:30:39.481993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114379+0000","last_change":"2024-03-16T14:31:06.703133+0000","last_active":"2024-03-16T14:31:45.114379+0000","last_peered":"2024-03-16T14:31:45.114379+0000","last_clean":"2024-03-16T14:31:45.114379+0000","last_became_active":"2024-03-16T14:31:06.702533+0000","last_became_peered":"2024-03-16T14:31:06.702533+0000","last_unstale":"2024-03-16T14:31:45.114379+0000","last_undegraded":"2024-03-16T14:31:45.114379+0000","last_fullsized":"2024-03-16T14:31:45.114379+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:51:06.809872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.122340+0000","last_change":"2024-03-16T14:31:10.721796+0000","last_active":"2024-03-16T14:31:45.122340+0000","last_peered":"2024-03-16T14:31:45.122340+0000","last_clean":"2024-03-16T14:31:45.122340+0000","last_became_active":"2024-03-16T14:31:10.721486+0000","last_became_peered":"2024-03-16T14:31:10.721486+0000","last_unstale":"2024-03-16T14:31:45.122340+0000","last_undegraded":"2024-03-16T14:31:45.122340+0000","last_fullsized":"2024-03-16T14:31:45.122340+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:12:42.239455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270861+0000","last_change":"2024-03-16T14:31:08.708164+0000","last_active":"2024-03-16T14:31:45.270861+0000","last_peered":"2024-03-16T14:31:45.270861+0000","last_clean":"2024-03-16T14:31:45.270861+0000","last_became_active":"2024-03-16T14:31:08.707311+0000","last_became_peered":"2024-03-16T14:31:08.707311+0000","last_unstale":"2024-03-16T14:31:45.270861+0000","last_undegraded":"2024-03-16T14:31:45.270861+0000","last_fullsized":"2024-03-16T14:31:45.270861+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:24:35.988978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128660+0000","last_change":"2024-03-16T14:31:04.677705+0000","last_active":"2024-03-16T14:31:45.128660+0000","last_peered":"2024-03-16T14:31:45.128660+0000","last_clean":"2024-03-16T14:31:45.128660+0000","last_became_active":"2024-03-16T14:31:04.677379+0000","last_became_peered":"2024-03-16T14:31:04.677379+0000","last_unstale":"2024-03-16T14:31:45.128660+0000","last_undegraded":"2024-03-16T14:31:45.128660+0000","last_fullsized":"2024-03-16T14:31:45.128660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:58:28.257779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011380+0000","last_change":"2024-03-16T14:31:06.702706+0000","last_active":"2024-03-16T14:31:45.011380+0000","last_peered":"2024-03-16T14:31:45.011380+0000","last_clean":"2024-03-16T14:31:45.011380+0000","last_became_active":"2024-03-16T14:31:06.702377+0000","last_became_peered":"2024-03-16T14:31:06.702377+0000","last_unstale":"2024-03-16T14:31:45.011380+0000","last_undegraded":"2024-03-16T14:31:45.011380+0000","last_fullsized":"2024-03-16T14:31:45.011380+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:11:22.020315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128921+0000","last_change":"2024-03-16T14:31:11.204053+0000","last_active":"2024-03-16T14:31:45.128921+0000","last_peered":"2024-03-16T14:31:45.128921+0000","last_clean":"2024-03-16T14:31:45.128921+0000","last_became_active":"2024-03-16T14:31:11.203738+0000","last_became_peered":"2024-03-16T14:31:11.203738+0000","last_unstale":"2024-03-16T14:31:45.128921+0000","last_undegraded":"2024-03-16T14:31:45.128921+0000","last_fullsized":"2024-03-16T14:31:45.128921+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:28:18.748789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114131+0000","last_change":"2024-03-16T14:31:08.709074+0000","last_active":"2024-03-16T14:31:45.114131+0000","last_peered":"2024-03-16T14:31:45.114131+0000","last_clean":"2024-03-16T14:31:45.114131+0000","last_became_active":"2024-03-16T14:31:08.708778+0000","last_became_peered":"2024-03-16T14:31:08.708778+0000","last_unstale":"2024-03-16T14:31:45.114131+0000","last_undegraded":"2024-03-16T14:31:45.114131+0000","last_fullsized":"2024-03-16T14:31:45.114131+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:21:36.373710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268271+0000","last_change":"2024-03-16T14:31:04.668540+0000","last_active":"2024-03-16T14:31:45.268271+0000","last_peered":"2024-03-16T14:31:45.268271+0000","last_clean":"2024-03-16T14:31:45.268271+0000","last_became_active":"2024-03-16T14:31:04.668221+0000","last_became_peered":"2024-03-16T14:31:04.668221+0000","last_unstale":"2024-03-16T14:31:45.268271+0000","last_undegraded":"2024-03-16T14:31:45.268271+0000","last_fullsized":"2024-03-16T14:31:45.268271+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:10:00.925985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'6","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270966+0000","last_change":"2024-03-16T14:31:06.696810+0000","last_active":"2024-03-16T14:31:45.270966+0000","last_peered":"2024-03-16T14:31:45.270966+0000","last_clean":"2024-03-16T14:31:45.270966+0000","last_became_active":"2024-03-16T14:31:06.696459+0000","last_became_peered":"2024-03-16T14:31:06.696459+0000","last_unstale":"2024-03-16T14:31:45.270966+0000","last_undegraded":"2024-03-16T14:31:45.270966+0000","last_fullsized":"2024-03-16T14:31:45.270966+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:36:11.615069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114187+0000","last_change":"2024-03-16T14:31:11.202128+0000","last_active":"2024-03-16T14:31:45.114187+0000","last_peered":"2024-03-16T14:31:45.114187+0000","last_clean":"2024-03-16T14:31:45.114187+0000","last_became_active":"2024-03-16T14:31:11.201786+0000","last_became_peered":"2024-03-16T14:31:11.201786+0000","last_unstale":"2024-03-16T14:31:45.114187+0000","last_undegraded":"2024-03-16T14:31:45.114187+0000","last_fullsized":"2024-03-16T14:31:45.114187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:15:29.379451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129172+0000","last_change":"2024-03-16T14:31:04.679710+0000","last_active":"2024-03-16T14:31:45.129172+0000","last_peered":"2024-03-16T14:31:45.129172+0000","last_clean":"2024-03-16T14:31:45.129172+0000","last_became_active":"2024-03-16T14:31:04.679433+0000","last_became_peered":"2024-03-16T14:31:04.679433+0000","last_unstale":"2024-03-16T14:31:45.129172+0000","last_undegraded":"2024-03-16T14:31:45.129172+0000","last_fullsized":"2024-03-16T14:31:45.129172+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:10:27.002679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113701+0000","last_change":"2024-03-16T14:31:06.701063+0000","last_active":"2024-03-16T14:31:45.113701+0000","last_peered":"2024-03-16T14:31:45.113701+0000","last_clean":"2024-03-16T14:31:45.113701+0000","last_became_active":"2024-03-16T14:31:06.700738+0000","last_became_peered":"2024-03-16T14:31:06.700738+0000","last_unstale":"2024-03-16T14:31:45.113701+0000","last_undegraded":"2024-03-16T14:31:45.113701+0000","last_fullsized":"2024-03-16T14:31:45.113701+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:57:13.931074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734560+0000","last_change":"2024-03-16T14:31:08.708111+0000","last_active":"2024-03-16T14:32:19.734560+0000","last_peered":"2024-03-16T14:32:19.734560+0000","last_clean":"2024-03-16T14:32:19.734560+0000","last_became_active":"2024-03-16T14:31:08.707770+0000","last_became_peered":"2024-03-16T14:31:08.707770+0000","last_unstale":"2024-03-16T14:32:19.734560+0000","last_undegraded":"2024-03-16T14:32:19.734560+0000","last_fullsized":"2024-03-16T14:32:19.734560+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:57:18.898584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270011+0000","last_change":"2024-03-16T14:31:10.724541+0000","last_active":"2024-03-16T14:31:45.270011+0000","last_peered":"2024-03-16T14:31:45.270011+0000","last_clean":"2024-03-16T14:31:45.270011+0000","last_became_active":"2024-03-16T14:31:10.724142+0000","last_became_peered":"2024-03-16T14:31:10.724142+0000","last_unstale":"2024-03-16T14:31:45.270011+0000","last_undegraded":"2024-03-16T14:31:45.270011+0000","last_fullsized":"2024-03-16T14:31:45.270011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:03:35.534603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129121+0000","last_change":"2024-03-16T14:31:04.678562+0000","last_active":"2024-03-16T14:31:45.129121+0000","last_peered":"2024-03-16T14:31:45.129121+0000","last_clean":"2024-03-16T14:31:45.129121+0000","last_became_active":"2024-03-16T14:31:04.678266+0000","last_became_peered":"2024-03-16T14:31:04.678266+0000","last_unstale":"2024-03-16T14:31:45.129121+0000","last_undegraded":"2024-03-16T14:31:45.129121+0000","last_fullsized":"2024-03-16T14:31:45.129121+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:01:44.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'12","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008308+0000","last_change":"2024-03-16T14:31:06.704081+0000","last_active":"2024-03-16T14:31:45.008308+0000","last_peered":"2024-03-16T14:31:45.008308+0000","last_clean":"2024-03-16T14:31:45.008308+0000","last_became_active":"2024-03-16T14:31:06.703762+0000","last_became_peered":"2024-03-16T14:31:06.703762+0000","last_unstale":"2024-03-16T14:31:45.008308+0000","last_undegraded":"2024-03-16T14:31:45.008308+0000","last_fullsized":"2024-03-16T14:31:45.008308+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:27:47.127996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121293+0000","last_change":"2024-03-16T14:31:08.710814+0000","last_active":"2024-03-16T14:31:45.121293+0000","last_peered":"2024-03-16T14:31:45.121293+0000","last_clean":"2024-03-16T14:31:45.121293+0000","last_became_active":"2024-03-16T14:31:08.710314+0000","last_became_peered":"2024-03-16T14:31:08.710314+0000","last_unstale":"2024-03-16T14:31:45.121293+0000","last_undegraded":"2024-03-16T14:31:45.121293+0000","last_fullsized":"2024-03-16T14:31:45.121293+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:53:31.627429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.128467+0000","last_change":"2024-03-16T14:31:10.725143+0000","last_active":"2024-03-16T14:31:45.128467+0000","last_peered":"2024-03-16T14:31:45.128467+0000","last_clean":"2024-03-16T14:31:45.128467+0000","last_became_active":"2024-03-16T14:31:10.724860+0000","last_became_peered":"2024-03-16T14:31:10.724860+0000","last_unstale":"2024-03-16T14:31:45.128467+0000","last_undegraded":"2024-03-16T14:31:45.128467+0000","last_fullsized":"2024-03-16T14:31:45.128467+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:40:09.584785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121552+0000","last_change":"2024-03-16T14:31:08.707272+0000","last_active":"2024-03-16T14:31:45.121552+0000","last_peered":"2024-03-16T14:31:45.121552+0000","last_clean":"2024-03-16T14:31:45.121552+0000","last_became_active":"2024-03-16T14:31:08.706314+0000","last_became_peered":"2024-03-16T14:31:08.706314+0000","last_unstale":"2024-03-16T14:31:45.121552+0000","last_undegraded":"2024-03-16T14:31:45.121552+0000","last_fullsized":"2024-03-16T14:31:45.121552+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:07:34.998482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129298+0000","last_change":"2024-03-16T14:31:04.672399+0000","last_active":"2024-03-16T14:31:45.129298+0000","last_peered":"2024-03-16T14:31:45.129298+0000","last_clean":"2024-03-16T14:31:45.129298+0000","last_became_active":"2024-03-16T14:31:04.672092+0000","last_became_peered":"2024-03-16T14:31:04.672092+0000","last_unstale":"2024-03-16T14:31:45.129298+0000","last_undegraded":"2024-03-16T14:31:45.129298+0000","last_fullsized":"2024-03-16T14:31:45.129298+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:30:55.442342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'19","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268696+0000","last_change":"2024-03-16T14:31:06.705167+0000","last_active":"2024-03-16T14:31:45.268696+0000","last_peered":"2024-03-16T14:31:45.268696+0000","last_clean":"2024-03-16T14:31:45.268696+0000","last_became_active":"2024-03-16T14:31:06.704852+0000","last_became_peered":"2024-03-16T14:31:06.704852+0000","last_unstale":"2024-03-16T14:31:45.268696+0000","last_undegraded":"2024-03-16T14:31:45.268696+0000","last_fullsized":"2024-03-16T14:31:45.268696+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:48:22.378927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008482+0000","last_change":"2024-03-16T14:31:10.721287+0000","last_active":"2024-03-16T14:31:45.008482+0000","last_peered":"2024-03-16T14:31:45.008482+0000","last_clean":"2024-03-16T14:31:45.008482+0000","last_became_active":"2024-03-16T14:31:10.720939+0000","last_became_peered":"2024-03-16T14:31:10.720939+0000","last_unstale":"2024-03-16T14:31:45.008482+0000","last_undegraded":"2024-03-16T14:31:45.008482+0000","last_fullsized":"2024-03-16T14:31:45.008482+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:47:00.942919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121461+0000","last_change":"2024-03-16T14:31:08.708168+0000","last_active":"2024-03-16T14:31:45.121461+0000","last_peered":"2024-03-16T14:31:45.121461+0000","last_clean":"2024-03-16T14:31:45.121461+0000","last_became_active":"2024-03-16T14:31:08.707780+0000","last_became_peered":"2024-03-16T14:31:08.707780+0000","last_unstale":"2024-03-16T14:31:45.121461+0000","last_undegraded":"2024-03-16T14:31:45.121461+0000","last_fullsized":"2024-03-16T14:31:45.121461+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:33.833906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114023+0000","last_change":"2024-03-16T14:31:04.670724+0000","last_active":"2024-03-16T14:31:45.114023+0000","last_peered":"2024-03-16T14:31:45.114023+0000","last_clean":"2024-03-16T14:31:45.114023+0000","last_became_active":"2024-03-16T14:31:04.670432+0000","last_became_peered":"2024-03-16T14:31:04.670432+0000","last_unstale":"2024-03-16T14:31:45.114023+0000","last_undegraded":"2024-03-16T14:31:45.114023+0000","last_fullsized":"2024-03-16T14:31:45.114023+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:53:51.765443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011367+0000","last_change":"2024-03-16T14:31:06.698773+0000","last_active":"2024-03-16T14:31:45.011367+0000","last_peered":"2024-03-16T14:31:45.011367+0000","last_clean":"2024-03-16T14:31:45.011367+0000","last_became_active":"2024-03-16T14:31:06.698472+0000","last_became_peered":"2024-03-16T14:31:06.698472+0000","last_unstale":"2024-03-16T14:31:45.011367+0000","last_undegraded":"2024-03-16T14:31:45.011367+0000","last_fullsized":"2024-03-16T14:31:45.011367+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:48.174804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270468+0000","last_change":"2024-03-16T14:31:10.719141+0000","last_active":"2024-03-16T14:31:45.270468+0000","last_peered":"2024-03-16T14:31:45.270468+0000","last_clean":"2024-03-16T14:31:45.270468+0000","last_became_active":"2024-03-16T14:31:10.718652+0000","last_became_peered":"2024-03-16T14:31:10.718652+0000","last_unstale":"2024-03-16T14:31:45.270468+0000","last_undegraded":"2024-03-16T14:31:45.270468+0000","last_fullsized":"2024-03-16T14:31:45.270468+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:55:08.345560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.113804+0000","last_change":"2024-03-16T14:31:08.701635+0000","last_active":"2024-03-16T14:31:45.113804+0000","last_peered":"2024-03-16T14:31:45.113804+0000","last_clean":"2024-03-16T14:31:45.113804+0000","last_became_active":"2024-03-16T14:31:04.674571+0000","last_became_peered":"2024-03-16T14:31:04.674571+0000","last_unstale":"2024-03-16T14:31:45.113804+0000","last_undegraded":"2024-03-16T14:31:45.113804+0000","last_fullsized":"2024-03-16T14:31:45.113804+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:54:41.218336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012146240000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008879+0000","last_change":"2024-03-16T14:31:04.669970+0000","last_active":"2024-03-16T14:31:45.008879+0000","last_peered":"2024-03-16T14:31:45.008879+0000","last_clean":"2024-03-16T14:31:45.008879+0000","last_became_active":"2024-03-16T14:31:04.669674+0000","last_became_peered":"2024-03-16T14:31:04.669674+0000","last_unstale":"2024-03-16T14:31:45.008879+0000","last_undegraded":"2024-03-16T14:31:45.008879+0000","last_fullsized":"2024-03-16T14:31:45.008879+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:10:54.842931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'28","reported_seq":94,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734518+0000","last_change":"2024-03-16T14:31:06.693402+0000","last_active":"2024-03-16T14:32:19.734518+0000","last_peered":"2024-03-16T14:32:19.734518+0000","last_clean":"2024-03-16T14:32:19.734518+0000","last_became_active":"2024-03-16T14:31:06.693065+0000","last_became_peered":"2024-03-16T14:31:06.693065+0000","last_unstale":"2024-03-16T14:32:19.734518+0000","last_undegraded":"2024-03-16T14:32:19.734518+0000","last_fullsized":"2024-03-16T14:32:19.734518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:33:54.615938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010780+0000","last_change":"2024-03-16T14:31:08.714945+0000","last_active":"2024-03-16T14:31:45.010780+0000","last_peered":"2024-03-16T14:31:45.010780+0000","last_clean":"2024-03-16T14:31:45.010780+0000","last_became_active":"2024-03-16T14:31:08.714637+0000","last_became_peered":"2024-03-16T14:31:08.714637+0000","last_unstale":"2024-03-16T14:31:45.010780+0000","last_undegraded":"2024-03-16T14:31:45.010780+0000","last_fullsized":"2024-03-16T14:31:45.010780+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:45:31.769929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129354+0000","last_change":"2024-03-16T14:31:11.202130+0000","last_active":"2024-03-16T14:31:45.129354+0000","last_peered":"2024-03-16T14:31:45.129354+0000","last_clean":"2024-03-16T14:31:45.129354+0000","last_became_active":"2024-03-16T14:31:11.201832+0000","last_became_peered":"2024-03-16T14:31:11.201832+0000","last_unstale":"2024-03-16T14:31:45.129354+0000","last_undegraded":"2024-03-16T14:31:45.129354+0000","last_fullsized":"2024-03-16T14:31:45.129354+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:57:11.276429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268713+0000","last_change":"2024-03-16T14:31:04.679021+0000","last_active":"2024-03-16T14:31:45.268713+0000","last_peered":"2024-03-16T14:31:45.268713+0000","last_clean":"2024-03-16T14:31:45.268713+0000","last_became_active":"2024-03-16T14:31:04.678786+0000","last_became_peered":"2024-03-16T14:31:04.678786+0000","last_unstale":"2024-03-16T14:31:45.268713+0000","last_undegraded":"2024-03-16T14:31:45.268713+0000","last_fullsized":"2024-03-16T14:31:45.268713+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:44:22.898738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121623+0000","last_change":"2024-03-16T14:31:08.695385+0000","last_active":"2024-03-16T14:31:45.121623+0000","last_peered":"2024-03-16T14:31:45.121623+0000","last_clean":"2024-03-16T14:31:45.121623+0000","last_became_active":"2024-03-16T14:31:04.671086+0000","last_became_peered":"2024-03-16T14:31:04.671086+0000","last_unstale":"2024-03-16T14:31:45.121623+0000","last_undegraded":"2024-03-16T14:31:45.121623+0000","last_fullsized":"2024-03-16T14:31:45.121623+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:45:23.153102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077491400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'13","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270395+0000","last_change":"2024-03-16T14:31:06.697643+0000","last_active":"2024-03-16T14:31:45.270395+0000","last_peered":"2024-03-16T14:31:45.270395+0000","last_clean":"2024-03-16T14:31:45.270395+0000","last_became_active":"2024-03-16T14:31:06.696749+0000","last_became_peered":"2024-03-16T14:31:06.696749+0000","last_unstale":"2024-03-16T14:31:45.270395+0000","last_undegraded":"2024-03-16T14:31:45.270395+0000","last_fullsized":"2024-03-16T14:31:45.270395+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T19:18:55.278292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121950+0000","last_change":"2024-03-16T14:31:08.709870+0000","last_active":"2024-03-16T14:31:45.121950+0000","last_peered":"2024-03-16T14:31:45.121950+0000","last_clean":"2024-03-16T14:31:45.121950+0000","last_became_active":"2024-03-16T14:31:08.709533+0000","last_became_peered":"2024-03-16T14:31:08.709533+0000","last_unstale":"2024-03-16T14:31:45.121950+0000","last_undegraded":"2024-03-16T14:31:45.121950+0000","last_fullsized":"2024-03-16T14:31:45.121950+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:00:08.155629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114327+0000","last_change":"2024-03-16T14:31:10.724387+0000","last_active":"2024-03-16T14:31:45.114327+0000","last_peered":"2024-03-16T14:31:45.114327+0000","last_clean":"2024-03-16T14:31:45.114327+0000","last_became_active":"2024-03-16T14:31:10.724048+0000","last_became_peered":"2024-03-16T14:31:10.724048+0000","last_unstale":"2024-03-16T14:31:45.114327+0000","last_undegraded":"2024-03-16T14:31:45.114327+0000","last_fullsized":"2024-03-16T14:31:45.114327+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:26:26.403169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":345,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:47.176163+0000","last_change":"2024-03-16T14:30:55.235452+0000","last_active":"2024-03-16T14:31:47.176163+0000","last_peered":"2024-03-16T14:31:47.176163+0000","last_clean":"2024-03-16T14:31:47.176163+0000","last_became_active":"2024-03-16T14:30:55.235047+0000","last_became_peered":"2024-03-16T14:30:55.235047+0000","last_unstale":"2024-03-16T14:31:47.176163+0000","last_undegraded":"2024-03-16T14:31:47.176163+0000","last_fullsized":"2024-03-16T14:31:47.176163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:28:44.874450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:28:44.874450+0000","last_clean_scrub_stamp":"2024-03-16T14:28:44.874450+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:11:13.723888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269239+0000","last_change":"2024-03-16T14:31:04.670629+0000","last_active":"2024-03-16T14:31:45.269239+0000","last_peered":"2024-03-16T14:31:45.269239+0000","last_clean":"2024-03-16T14:31:45.269239+0000","last_became_active":"2024-03-16T14:31:04.670344+0000","last_became_peered":"2024-03-16T14:31:04.670344+0000","last_unstale":"2024-03-16T14:31:45.269239+0000","last_undegraded":"2024-03-16T14:31:45.269239+0000","last_fullsized":"2024-03-16T14:31:45.269239+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:41:48.580850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734577+0000","last_change":"2024-03-16T14:31:06.698898+0000","last_active":"2024-03-16T14:32:19.734577+0000","last_peered":"2024-03-16T14:32:19.734577+0000","last_clean":"2024-03-16T14:32:19.734577+0000","last_became_active":"2024-03-16T14:31:06.698386+0000","last_became_peered":"2024-03-16T14:31:06.698386+0000","last_unstale":"2024-03-16T14:32:19.734577+0000","last_undegraded":"2024-03-16T14:32:19.734577+0000","last_fullsized":"2024-03-16T14:32:19.734577+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:20:54.053320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129026+0000","last_change":"2024-03-16T14:31:08.714519+0000","last_active":"2024-03-16T14:31:45.129026+0000","last_peered":"2024-03-16T14:31:45.129026+0000","last_clean":"2024-03-16T14:31:45.129026+0000","last_became_active":"2024-03-16T14:31:08.713445+0000","last_became_peered":"2024-03-16T14:31:08.713445+0000","last_unstale":"2024-03-16T14:31:45.129026+0000","last_undegraded":"2024-03-16T14:31:45.129026+0000","last_fullsized":"2024-03-16T14:31:45.129026+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:57:50.671228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269772+0000","last_change":"2024-03-16T14:31:10.720846+0000","last_active":"2024-03-16T14:31:45.269772+0000","last_peered":"2024-03-16T14:31:45.269772+0000","last_clean":"2024-03-16T14:31:45.269772+0000","last_became_active":"2024-03-16T14:31:10.720513+0000","last_became_peered":"2024-03-16T14:31:10.720513+0000","last_unstale":"2024-03-16T14:31:45.269772+0000","last_undegraded":"2024-03-16T14:31:45.269772+0000","last_fullsized":"2024-03-16T14:31:45.269772+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:02:26.456923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008708+0000","last_change":"2024-03-16T14:31:04.664536+0000","last_active":"2024-03-16T14:31:45.008708+0000","last_peered":"2024-03-16T14:31:45.008708+0000","last_clean":"2024-03-16T14:31:45.008708+0000","last_became_active":"2024-03-16T14:31:04.664242+0000","last_became_peered":"2024-03-16T14:31:04.664242+0000","last_unstale":"2024-03-16T14:31:45.008708+0000","last_undegraded":"2024-03-16T14:31:45.008708+0000","last_fullsized":"2024-03-16T14:31:45.008708+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:01:05.142473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":118,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:24.778005+0000","last_change":"2024-03-16T14:31:08.701212+0000","last_active":"2024-03-16T14:32:24.778005+0000","last_peered":"2024-03-16T14:32:24.778005+0000","last_clean":"2024-03-16T14:32:24.778005+0000","last_became_active":"2024-03-16T14:31:04.680416+0000","last_became_peered":"2024-03-16T14:31:04.680416+0000","last_unstale":"2024-03-16T14:32:24.778005+0000","last_undegraded":"2024-03-16T14:32:24.778005+0000","last_fullsized":"2024-03-16T14:32:24.778005+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:48:48.659045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057932800000000005,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011021+0000","last_change":"2024-03-16T14:31:32.535936+0000","last_active":"2024-03-16T14:31:45.011021+0000","last_peered":"2024-03-16T14:31:45.011021+0000","last_clean":"2024-03-16T14:31:45.011021+0000","last_became_active":"2024-03-16T14:31:32.528406+0000","last_became_peered":"2024-03-16T14:31:32.528406+0000","last_unstale":"2024-03-16T14:31:45.011021+0000","last_undegraded":"2024-03-16T14:31:45.011021+0000","last_fullsized":"2024-03-16T14:31:45.011021+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:11:10.755811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114541+0000","last_change":"2024-03-16T14:31:08.710092+0000","last_active":"2024-03-16T14:31:45.114541+0000","last_peered":"2024-03-16T14:31:45.114541+0000","last_clean":"2024-03-16T14:31:45.114541+0000","last_became_active":"2024-03-16T14:31:08.709756+0000","last_became_peered":"2024-03-16T14:31:08.709756+0000","last_unstale":"2024-03-16T14:31:45.114541+0000","last_undegraded":"2024-03-16T14:31:45.114541+0000","last_fullsized":"2024-03-16T14:31:45.114541+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:32:39.637196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268946+0000","last_change":"2024-03-16T14:31:11.202267+0000","last_active":"2024-03-16T14:31:45.268946+0000","last_peered":"2024-03-16T14:31:45.268946+0000","last_clean":"2024-03-16T14:31:45.268946+0000","last_became_active":"2024-03-16T14:31:11.201828+0000","last_became_peered":"2024-03-16T14:31:11.201828+0000","last_unstale":"2024-03-16T14:31:45.268946+0000","last_undegraded":"2024-03-16T14:31:45.268946+0000","last_fullsized":"2024-03-16T14:31:45.268946+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:00:57.106478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269971+0000","last_change":"2024-03-16T14:31:04.676172+0000","last_active":"2024-03-16T14:31:45.269971+0000","last_peered":"2024-03-16T14:31:45.269971+0000","last_clean":"2024-03-16T14:31:45.269971+0000","last_became_active":"2024-03-16T14:31:04.675680+0000","last_became_peered":"2024-03-16T14:31:04.675680+0000","last_unstale":"2024-03-16T14:31:45.269971+0000","last_undegraded":"2024-03-16T14:31:45.269971+0000","last_fullsized":"2024-03-16T14:31:45.269971+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:45:45.937214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'19","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011183+0000","last_change":"2024-03-16T14:31:06.697679+0000","last_active":"2024-03-16T14:31:45.011183+0000","last_peered":"2024-03-16T14:31:45.011183+0000","last_clean":"2024-03-16T14:31:45.011183+0000","last_became_active":"2024-03-16T14:31:06.697353+0000","last_became_peered":"2024-03-16T14:31:06.697353+0000","last_unstale":"2024-03-16T14:31:45.011183+0000","last_undegraded":"2024-03-16T14:31:45.011183+0000","last_fullsized":"2024-03-16T14:31:45.011183+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T17:49:06.836585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268797+0000","last_change":"2024-03-16T14:31:08.710342+0000","last_active":"2024-03-16T14:31:45.268797+0000","last_peered":"2024-03-16T14:31:45.268797+0000","last_clean":"2024-03-16T14:31:45.268797+0000","last_became_active":"2024-03-16T14:31:08.710005+0000","last_became_peered":"2024-03-16T14:31:08.710005+0000","last_unstale":"2024-03-16T14:31:45.268797+0000","last_undegraded":"2024-03-16T14:31:45.268797+0000","last_fullsized":"2024-03-16T14:31:45.268797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:07:49.614300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008350+0000","last_change":"2024-03-16T14:31:10.723297+0000","last_active":"2024-03-16T14:31:45.008350+0000","last_peered":"2024-03-16T14:31:45.008350+0000","last_clean":"2024-03-16T14:31:45.008350+0000","last_became_active":"2024-03-16T14:31:10.722665+0000","last_became_peered":"2024-03-16T14:31:10.722665+0000","last_unstale":"2024-03-16T14:31:45.008350+0000","last_undegraded":"2024-03-16T14:31:45.008350+0000","last_fullsized":"2024-03-16T14:31:45.008350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:47:25.111344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114257+0000","last_change":"2024-03-16T14:31:04.671435+0000","last_active":"2024-03-16T14:31:45.114257+0000","last_peered":"2024-03-16T14:31:45.114257+0000","last_clean":"2024-03-16T14:31:45.114257+0000","last_became_active":"2024-03-16T14:31:04.671132+0000","last_became_peered":"2024-03-16T14:31:04.671132+0000","last_unstale":"2024-03-16T14:31:45.114257+0000","last_undegraded":"2024-03-16T14:31:45.114257+0000","last_fullsized":"2024-03-16T14:31:45.114257+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:05:53.732301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269509+0000","last_change":"2024-03-16T14:31:06.696481+0000","last_active":"2024-03-16T14:31:45.269509+0000","last_peered":"2024-03-16T14:31:45.269509+0000","last_clean":"2024-03-16T14:31:45.269509+0000","last_became_active":"2024-03-16T14:31:06.696124+0000","last_became_peered":"2024-03-16T14:31:06.696124+0000","last_unstale":"2024-03-16T14:31:45.269509+0000","last_undegraded":"2024-03-16T14:31:45.269509+0000","last_fullsized":"2024-03-16T14:31:45.269509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T20:29:07.025616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734552+0000","last_change":"2024-03-16T14:31:08.714259+0000","last_active":"2024-03-16T14:32:19.734552+0000","last_peered":"2024-03-16T14:32:19.734552+0000","last_clean":"2024-03-16T14:32:19.734552+0000","last_became_active":"2024-03-16T14:31:08.713926+0000","last_became_peered":"2024-03-16T14:31:08.713926+0000","last_unstale":"2024-03-16T14:32:19.734552+0000","last_undegraded":"2024-03-16T14:32:19.734552+0000","last_fullsized":"2024-03-16T14:32:19.734552+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:30:42.410143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.129145+0000","last_change":"2024-03-16T14:31:11.204673+0000","last_active":"2024-03-16T14:31:45.129145+0000","last_peered":"2024-03-16T14:31:45.129145+0000","last_clean":"2024-03-16T14:31:45.129145+0000","last_became_active":"2024-03-16T14:31:11.204349+0000","last_became_peered":"2024-03-16T14:31:11.204349+0000","last_unstale":"2024-03-16T14:31:45.129145+0000","last_undegraded":"2024-03-16T14:31:45.129145+0000","last_fullsized":"2024-03-16T14:31:45.129145+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T14:49:56.361934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.011154+0000","last_change":"2024-03-16T14:31:04.675031+0000","last_active":"2024-03-16T14:31:45.011154+0000","last_peered":"2024-03-16T14:31:45.011154+0000","last_clean":"2024-03-16T14:31:45.011154+0000","last_became_active":"2024-03-16T14:31:04.674382+0000","last_became_peered":"2024-03-16T14:31:04.674382+0000","last_unstale":"2024-03-16T14:31:45.011154+0000","last_undegraded":"2024-03-16T14:31:45.011154+0000","last_fullsized":"2024-03-16T14:31:45.011154+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:52:16.718987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'14","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008219+0000","last_change":"2024-03-16T14:31:06.697569+0000","last_active":"2024-03-16T14:31:45.008219+0000","last_peered":"2024-03-16T14:31:45.008219+0000","last_clean":"2024-03-16T14:31:45.008219+0000","last_became_active":"2024-03-16T14:31:06.697220+0000","last_became_peered":"2024-03-16T14:31:06.697220+0000","last_unstale":"2024-03-16T14:31:45.008219+0000","last_undegraded":"2024-03-16T14:31:45.008219+0000","last_fullsized":"2024-03-16T14:31:45.008219+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:42:25.829258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734509+0000","last_change":"2024-03-16T14:31:08.712851+0000","last_active":"2024-03-16T14:32:19.734509+0000","last_peered":"2024-03-16T14:32:19.734509+0000","last_clean":"2024-03-16T14:32:19.734509+0000","last_became_active":"2024-03-16T14:31:08.712514+0000","last_became_peered":"2024-03-16T14:31:08.712514+0000","last_unstale":"2024-03-16T14:32:19.734509+0000","last_undegraded":"2024-03-16T14:32:19.734509+0000","last_fullsized":"2024-03-16T14:32:19.734509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:47:34.072218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008389+0000","last_change":"2024-03-16T14:31:10.716362+0000","last_active":"2024-03-16T14:31:45.008389+0000","last_peered":"2024-03-16T14:31:45.008389+0000","last_clean":"2024-03-16T14:31:45.008389+0000","last_became_active":"2024-03-16T14:31:10.716038+0000","last_became_peered":"2024-03-16T14:31:10.716038+0000","last_unstale":"2024-03-16T14:31:45.008389+0000","last_undegraded":"2024-03-16T14:31:45.008389+0000","last_fullsized":"2024-03-16T14:31:45.008389+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:37:32.291747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270709+0000","last_change":"2024-03-16T14:31:04.670979+0000","last_active":"2024-03-16T14:31:45.270709+0000","last_peered":"2024-03-16T14:31:45.270709+0000","last_clean":"2024-03-16T14:31:45.270709+0000","last_became_active":"2024-03-16T14:31:04.670667+0000","last_became_peered":"2024-03-16T14:31:04.670667+0000","last_unstale":"2024-03-16T14:31:45.270709+0000","last_undegraded":"2024-03-16T14:31:45.270709+0000","last_fullsized":"2024-03-16T14:31:45.270709+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:58:15.354541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'18","reported_seq":61,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269073+0000","last_change":"2024-03-16T14:31:06.703641+0000","last_active":"2024-03-16T14:31:45.269073+0000","last_peered":"2024-03-16T14:31:45.269073+0000","last_clean":"2024-03-16T14:31:45.269073+0000","last_became_active":"2024-03-16T14:31:06.703234+0000","last_became_peered":"2024-03-16T14:31:06.703234+0000","last_unstale":"2024-03-16T14:31:45.269073+0000","last_undegraded":"2024-03-16T14:31:45.269073+0000","last_fullsized":"2024-03-16T14:31:45.269073+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:17:00.265402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.007993+0000","last_change":"2024-03-16T14:31:08.712346+0000","last_active":"2024-03-16T14:31:45.007993+0000","last_peered":"2024-03-16T14:31:45.007993+0000","last_clean":"2024-03-16T14:31:45.007993+0000","last_became_active":"2024-03-16T14:31:08.711176+0000","last_became_peered":"2024-03-16T14:31:08.711176+0000","last_unstale":"2024-03-16T14:31:45.007993+0000","last_undegraded":"2024-03-16T14:31:45.007993+0000","last_fullsized":"2024-03-16T14:31:45.007993+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:07:33.415351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114074+0000","last_change":"2024-03-16T14:31:10.719134+0000","last_active":"2024-03-16T14:31:45.114074+0000","last_peered":"2024-03-16T14:31:45.114074+0000","last_clean":"2024-03-16T14:31:45.114074+0000","last_became_active":"2024-03-16T14:31:10.718753+0000","last_became_peered":"2024-03-16T14:31:10.718753+0000","last_unstale":"2024-03-16T14:31:45.114074+0000","last_undegraded":"2024-03-16T14:31:45.114074+0000","last_fullsized":"2024-03-16T14:31:45.114074+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:53:58.407790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734561+0000","last_change":"2024-03-16T14:31:08.707754+0000","last_active":"2024-03-16T14:32:19.734561+0000","last_peered":"2024-03-16T14:32:19.734561+0000","last_clean":"2024-03-16T14:32:19.734561+0000","last_became_active":"2024-03-16T14:31:08.707408+0000","last_became_peered":"2024-03-16T14:31:08.707408+0000","last_unstale":"2024-03-16T14:32:19.734561+0000","last_undegraded":"2024-03-16T14:32:19.734561+0000","last_fullsized":"2024-03-16T14:32:19.734561+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:04:09.053922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268976+0000","last_change":"2024-03-16T14:31:04.677127+0000","last_active":"2024-03-16T14:31:45.268976+0000","last_peered":"2024-03-16T14:31:45.268976+0000","last_clean":"2024-03-16T14:31:45.268976+0000","last_became_active":"2024-03-16T14:31:04.676790+0000","last_became_peered":"2024-03-16T14:31:04.676790+0000","last_unstale":"2024-03-16T14:31:45.268976+0000","last_undegraded":"2024-03-16T14:31:45.268976+0000","last_fullsized":"2024-03-16T14:31:45.268976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:05:47.401589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270594+0000","last_change":"2024-03-16T14:31:06.699015+0000","last_active":"2024-03-16T14:31:45.270594+0000","last_peered":"2024-03-16T14:31:45.270594+0000","last_clean":"2024-03-16T14:31:45.270594+0000","last_became_active":"2024-03-16T14:31:06.698694+0000","last_became_peered":"2024-03-16T14:31:06.698694+0000","last_unstale":"2024-03-16T14:31:45.270594+0000","last_undegraded":"2024-03-16T14:31:45.270594+0000","last_fullsized":"2024-03-16T14:31:45.270594+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:11:36.443865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.269452+0000","last_change":"2024-03-16T14:31:11.203288+0000","last_active":"2024-03-16T14:31:45.269452+0000","last_peered":"2024-03-16T14:31:45.269452+0000","last_clean":"2024-03-16T14:31:45.269452+0000","last_became_active":"2024-03-16T14:31:11.201970+0000","last_became_peered":"2024-03-16T14:31:11.201970+0000","last_unstale":"2024-03-16T14:31:45.269452+0000","last_undegraded":"2024-03-16T14:31:45.269452+0000","last_fullsized":"2024-03-16T14:31:45.269452+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T15:56:39.992885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.114487+0000","last_change":"2024-03-16T14:31:08.711780+0000","last_active":"2024-03-16T14:31:45.114487+0000","last_peered":"2024-03-16T14:31:45.114487+0000","last_clean":"2024-03-16T14:31:45.114487+0000","last_became_active":"2024-03-16T14:31:08.710957+0000","last_became_peered":"2024-03-16T14:31:08.710957+0000","last_unstale":"2024-03-16T14:31:45.114487+0000","last_undegraded":"2024-03-16T14:31:45.114487+0000","last_fullsized":"2024-03-16T14:31:45.114487+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T02:01:56.398499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.008080+0000","last_change":"2024-03-16T14:31:04.672961+0000","last_active":"2024-03-16T14:31:45.008080+0000","last_peered":"2024-03-16T14:31:45.008080+0000","last_clean":"2024-03-16T14:31:45.008080+0000","last_became_active":"2024-03-16T14:31:04.672580+0000","last_became_peered":"2024-03-16T14:31:04.672580+0000","last_unstale":"2024-03-16T14:31:45.008080+0000","last_undegraded":"2024-03-16T14:31:45.008080+0000","last_fullsized":"2024-03-16T14:31:45.008080+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:39:27.841097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010104+0000","last_change":"2024-03-16T14:31:06.702337+0000","last_active":"2024-03-16T14:31:45.010104+0000","last_peered":"2024-03-16T14:31:45.010104+0000","last_clean":"2024-03-16T14:31:45.010104+0000","last_became_active":"2024-03-16T14:31:06.701895+0000","last_became_peered":"2024-03-16T14:31:06.701895+0000","last_unstale":"2024-03-16T14:31:45.010104+0000","last_undegraded":"2024-03-16T14:31:45.010104+0000","last_fullsized":"2024-03-16T14:31:45.010104+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:53:37.149233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.121878+0000","last_change":"2024-03-16T14:31:10.719020+0000","last_active":"2024-03-16T14:31:45.121878+0000","last_peered":"2024-03-16T14:31:45.121878+0000","last_clean":"2024-03-16T14:31:45.121878+0000","last_became_active":"2024-03-16T14:31:10.718666+0000","last_became_peered":"2024-03-16T14:31:10.718666+0000","last_unstale":"2024-03-16T14:31:45.121878+0000","last_undegraded":"2024-03-16T14:31:45.121878+0000","last_fullsized":"2024-03-16T14:31:45.121878+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T01:10:40.679127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270132+0000","last_change":"2024-03-16T14:31:08.716074+0000","last_active":"2024-03-16T14:31:45.270132+0000","last_peered":"2024-03-16T14:31:45.270132+0000","last_clean":"2024-03-16T14:31:45.270132+0000","last_became_active":"2024-03-16T14:31:08.715693+0000","last_became_peered":"2024-03-16T14:31:08.715693+0000","last_unstale":"2024-03-16T14:31:45.270132+0000","last_undegraded":"2024-03-16T14:31:45.270132+0000","last_fullsized":"2024-03-16T14:31:45.270132+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T23:29:07.054526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.268413+0000","last_change":"2024-03-16T14:31:04.674235+0000","last_active":"2024-03-16T14:31:45.268413+0000","last_peered":"2024-03-16T14:31:45.268413+0000","last_clean":"2024-03-16T14:31:45.268413+0000","last_became_active":"2024-03-16T14:31:04.673944+0000","last_became_peered":"2024-03-16T14:31:04.673944+0000","last_unstale":"2024-03-16T14:31:45.268413+0000","last_undegraded":"2024-03-16T14:31:45.268413+0000","last_fullsized":"2024-03-16T14:31:45.268413+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:00:09.832597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009683+0000","last_change":"2024-03-16T14:31:32.543771+0000","last_active":"2024-03-16T14:31:45.009683+0000","last_peered":"2024-03-16T14:31:45.009683+0000","last_clean":"2024-03-16T14:31:45.009683+0000","last_became_active":"2024-03-16T14:31:32.528585+0000","last_became_peered":"2024-03-16T14:31:32.528585+0000","last_unstale":"2024-03-16T14:31:45.009683+0000","last_undegraded":"2024-03-16T14:31:45.009683+0000","last_fullsized":"2024-03-16T14:31:45.009683+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T18:43:12.134675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.010353+0000","last_change":"2024-03-16T14:31:10.719838+0000","last_active":"2024-03-16T14:31:45.010353+0000","last_peered":"2024-03-16T14:31:45.010353+0000","last_clean":"2024-03-16T14:31:45.010353+0000","last_became_active":"2024-03-16T14:31:10.719337+0000","last_became_peered":"2024-03-16T14:31:10.719337+0000","last_unstale":"2024-03-16T14:31:45.010353+0000","last_undegraded":"2024-03-16T14:31:45.010353+0000","last_fullsized":"2024-03-16T14:31:45.010353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:09.692305+0000","last_clean_scrub_stamp":"2024-03-16T14:31:09.692305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T21:30:49.411444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:32:19.734515+0000","last_change":"2024-03-16T14:31:08.709343+0000","last_active":"2024-03-16T14:32:19.734515+0000","last_peered":"2024-03-16T14:32:19.734515+0000","last_clean":"2024-03-16T14:32:19.734515+0000","last_became_active":"2024-03-16T14:31:08.708152+0000","last_became_peered":"2024-03-16T14:31:08.708152+0000","last_unstale":"2024-03-16T14:32:19.734515+0000","last_undegraded":"2024-03-16T14:32:19.734515+0000","last_fullsized":"2024-03-16T14:32:19.734515+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:07.678661+0000","last_clean_scrub_stamp":"2024-03-16T14:31:07.678661+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-18T00:01:00.356461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.270278+0000","last_change":"2024-03-16T14:31:04.667057+0000","last_active":"2024-03-16T14:31:45.270278+0000","last_peered":"2024-03-16T14:31:45.270278+0000","last_clean":"2024-03-16T14:31:45.270278+0000","last_became_active":"2024-03-16T14:31:04.666730+0000","last_became_peered":"2024-03-16T14:31:04.666730+0000","last_unstale":"2024-03-16T14:31:45.270278+0000","last_undegraded":"2024-03-16T14:31:45.270278+0000","last_fullsized":"2024-03-16T14:31:45.270278+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:03.650418+0000","last_clean_scrub_stamp":"2024-03-16T14:31:03.650418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T22:05:47.841848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-16T14:31:45.009999+0000","last_change":"2024-03-16T14:31:06.702973+0000","last_active":"2024-03-16T14:31:45.009999+0000","last_peered":"2024-03-16T14:31:45.009999+0000","last_clean":"2024-03-16T14:31:45.009999+0000","last_became_active":"2024-03-16T14:31:06.702659+0000","last_became_peered":"2024-03-16T14:31:06.702659+0000","last_unstale":"2024-03-16T14:31:45.009999+0000","last_undegraded":"2024-03-16T14:31:45.009999+0000","last_fullsized":"2024-03-16T14:31:45.009999+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-16T14:31:05.666423+0000","last_clean_scrub_stamp":"2024-03-16T14:31:05.666423+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-17T16:19:22.664741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32540,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708516,"statfs":{"total":95990841344,"available":95957520384,"internally_reserved":0,"allocated":1392640,"data_stored":923102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28364,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712692,"statfs":{"total":95990841344,"available":95961796608,"internally_reserved":0,"allocated":1380352,"data_stored":923397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713224,"statfs":{"total":95990841344,"available":95962341376,"internally_reserved":0,"allocated":831488,"data_stored":333560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712680,"statfs":{"total":95990841344,"available":95961784320,"internally_reserved":0,"allocated":1392640,"data_stored":922831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31948,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709108,"statfs":{"total":95990841344,"available":95958126592,"internally_reserved":0,"allocated":786432,"data_stored":332453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713244,"statfs":{"total":95990841344,"available":95962361856,"internally_reserved":0,"allocated":811008,"data_stored":332560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31968,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709088,"statfs":{"total":95990841344,"available":95958106112,"internally_reserved":0,"allocated":806912,"data_stored":333185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713268,"statfs":{"total":95990841344,"available":95962386432,"internally_reserved":0,"allocated":786432,"data_stored":333016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-16T14:32:31.445 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-16T14:32:31.445 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-16T14:32:31.446 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-16T14:32:31.446 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-16T14:32:31.717 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:32.874 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:32 smithi026 ceph-mon[38287]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T14:32:32.874 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:32 smithi026 ceph-mon[38287]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:32.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:32 smithi026 ceph-mon[25218]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T14:32:32.874 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:32 smithi026 ceph-mon[25218]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:32.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:32 smithi151 ceph-mon[30603]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-16T14:32:32.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:32 smithi151 ceph-mon[30603]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:33.637 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-16T14:32:33.637 INFO:teuthology.orchestra.run.smithi026.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-16T14:32:34.365 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-16T14:32:34.365 INFO:tasks.cephadm:Setup complete, yielding 2024-03-16T14:32:34.366 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-16T14:32:34.376 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi026.front.sepia.ceph.com 2024-03-16T14:32:34.376 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-16T14:32:34.627 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:34.750 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:34 smithi026 ceph-mon[38287]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:34.750 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:34 smithi026 ceph-mon[38287]: from='client.? 172.21.15.26:0/2832083832' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T14:32:34.750 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:34 smithi026 ceph-mon[25218]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:34.751 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:34 smithi026 ceph-mon[25218]: from='client.? 172.21.15.26:0/2832083832' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T14:32:34.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:34 smithi151 ceph-mon[30603]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:34.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:34 smithi151 ceph-mon[30603]: from='client.? 172.21.15.26:0/2832083832' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-16T14:32:36.274 INFO:teuthology.orchestra.run.smithi026.stdout:167 167 2024-03-16T14:32:36.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:36 smithi151 ceph-mon[30603]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:36.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:32:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:32:37.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:36 smithi026 ceph-mon[38287]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:37.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:36 smithi026 ceph-mon[25218]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:37.232 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-16T14:32:37.483 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:37.769 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:32:37 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:32:37.516Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004255088s 2024-03-16T14:32:38.607 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:38 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:38.607 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:38 smithi026 ceph-mon[38287]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:38.608 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:38 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:38.608 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:38 smithi026 ceph-mon[25218]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:38.906 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:38 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:32:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:32:38.950 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:38 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:38.950 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:38 smithi151 ceph-mon[30603]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:39.527 INFO:teuthology.orchestra.run.smithi026.stdout:Backend: cephadm 2024-03-16T14:32:39.527 INFO:teuthology.orchestra.run.smithi026.stdout:Available: Yes 2024-03-16T14:32:39.528 INFO:teuthology.orchestra.run.smithi026.stdout:Paused: No 2024-03-16T14:32:40.239 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-16T14:32:40.487 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:40.638 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:40 smithi151 ceph-mon[30603]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:40.638 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:40 smithi151 ceph-mon[30603]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:40.837 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:40 smithi026 ceph-mon[25218]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:40.837 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:40 smithi026 ceph-mon[25218]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:40.837 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:40 smithi026 ceph-mon[38287]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:40.837 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:40 smithi026 ceph-mon[38287]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:42.088 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 podman[70826]: 2024-03-16 14:32:41.918668431 +0000 UTC m=+0.017410086 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-16T14:32:42.088 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 podman[70826]: 2024-03-16 14:32:42.086228342 +0000 UTC m=+0.184969983 container create 84254fe43d59625b19a891ea74d37ce1efa4b416bf0c5d1b5c11de50934080b0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a, release=239.1651231664, vendor=Red Hat, Inc., description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, maintainer=Paul Cuzner , architecture=x86_64, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, name=ubi8, version=8.5) 2024-03-16T14:32:42.500 INFO:teuthology.orchestra.run.smithi026.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-16T14:32:42.500 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager.a smithi026 *:9093,9094 starting - - - - 2024-03-16T14:32:42.500 INFO:teuthology.orchestra.run.smithi026.stdout:iscsi.iscsi.a smithi151 *:5000 running (86s) 53s ago 86s - - 3.7 3f3b48f8e85f c51e5d771428 2024-03-16T14:32:42.500 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.x smithi151 *:8443,8765 running (5m) 53s ago 5m - - 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 24f82a3b44e4 2024-03-16T14:32:42.500 INFO:teuthology.orchestra.run.smithi026.stdout:mgr.y smithi026 *:9283,8765 running (6m) 28s ago 6m 483M - 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f cfd474663860 2024-03-16T14:32:42.500 INFO:teuthology.orchestra.run.smithi026.stdout:mon.a smithi026 running (7m) 28s ago 7m 51.5M 2048M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 3b7c51472ad6 2024-03-16T14:32:42.500 INFO:teuthology.orchestra.run.smithi026.stdout:mon.b smithi151 running (5m) 53s ago 5m - 2048M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 10f194751dd6 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:mon.c smithi026 running (5m) 28s ago 5m 51.1M 2048M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 12f7db96a360 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.0 smithi026 running (4m) 28s ago 4m 45.6M 3761M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f b7a6b3dfa229 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.1 smithi026 running (4m) 28s ago 4m 46.4M 3761M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f ff7440a76d74 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.2 smithi026 running (4m) 28s ago 4m 51.8M 3761M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 5b3fbc0a4cd7 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.3 smithi026 running (3m) 28s ago 3m 42.7M 3761M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 8cfb446f63ee 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.4 smithi151 running (3m) 53s ago 3m - 3760M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 89ade8a73db0 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.5 smithi151 running (2m) 53s ago 2m - 3760M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 54b538e68a65 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.6 smithi151 running (2m) 53s ago 2m - 3760M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 15214c61d1d8 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:osd.7 smithi151 running (117s) 53s ago 116s - 3760M 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 384cce9e96b7 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus.a smithi151 *:9095 running (66s) 53s ago 66s - - 2.43.0 a07b618ecd1d ab86899811bd 2024-03-16T14:32:42.501 INFO:teuthology.orchestra.run.smithi026.stdout:rgw.foo.a smithi026 *:80 running (99s) 28s ago 99s 75.0M - 19.0.0-2121-g5ea4d9cf 3f3b48f8e85f 3c2ac519e932 2024-03-16T14:32:42.608 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-mon[30603]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:42.609 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:42.609 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 podman[70826]: 2024-03-16 14:32:42.356197554 +0000 UTC m=+0.454939203 container init 84254fe43d59625b19a891ea74d37ce1efa4b416bf0c5d1b5c11de50934080b0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, version=8.5, description=Ceph Grafana Container, io.buildah.version=1.32.0, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897) 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 podman[70826]: 2024-03-16 14:32:42.359529063 +0000 UTC m=+0.458270711 container start 84254fe43d59625b19a891ea74d37ce1efa4b416bf0c5d1b5c11de50934080b0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=ubi8, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container) 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 bash[70826]: 84254fe43d59625b19a891ea74d37ce1efa4b416bf0c5d1b5c11de50934080b0 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 systemd[1]: Started Ceph grafana.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.520963363Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521151619Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521170024Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-16T14:32:42.609 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521177658Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-16T14:32:42.610 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521183668Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-16T14:32:42.610 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521189664Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-16T14:32:42.610 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.52119523Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-16T14:32:42.610 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521202013Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-16T14:32:42.610 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521207722Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-16T14:32:42.610 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521213337Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-16T14:32:42.610 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521218703Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521224072Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521229579Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=settings t=2024-03-16T14:32:42.521235217Z level=info msg="App mode production" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=sqlstore t=2024-03-16T14:32:42.521277329Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=sqlstore t=2024-03-16T14:32:42.521293102Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.522134399Z level=info msg="Starting DB migrations" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.522207972Z level=info msg="Executing migration" id="create migration_log table" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.522725707Z level=info msg="Executing migration" id="create user table" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.523156662Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.525910461Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.528830645Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.529625492Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.532295382Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.535114735Z level=info msg="Executing migration" id="create user table v2" 2024-03-16T14:32:42.611 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.537787887Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.54057747Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.543398698Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.545946462Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.546514414Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.54923251Z level=info msg="Executing migration" id="Update user table charset" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.55172216Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.554412129Z level=info msg="Executing migration" id="Add missing user data" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.554894067Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.557694817Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.558137853Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.558553Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.559751273Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.5611926Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.566424331Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.567041816Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-16T14:32:42.612 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.567867993Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-16T14:32:42.613 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.569449304Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-16T14:32:42.613 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.569880772Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-16T14:32:42.613 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.57070718Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-16T14:32:42.613 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.571244863Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-16T14:32:42.613 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.571640491Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-16T14:32:42.613 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.572028308Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.572637419Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.573030888Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.573402844Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.57379381Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.574283363Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.575277236Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.576095905Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.576733428Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.577623288Z level=info msg="Executing migration" id="create star table" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.578536764Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.579005342Z level=info msg="Executing migration" id="create org table v1" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.579861872Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.58069398Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-16T14:32:42.614 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.581638802Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.582348693Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.583428719Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.584079968Z level=info msg="Executing migration" id="Update org table charset" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.584366799Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.584694244Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.585265764Z level=info msg="Executing migration" id="create dashboard table" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.586193778Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.587076811Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.587668293Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.588500289Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.589454188Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.590203413Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.591125923Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.591961245Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-16T14:32:42.615 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.595004721Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-16T14:32:42.616 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.597890739Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-16T14:32:42.616 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.598664783Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-16T14:32:42.616 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.601565813Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-16T14:32:42.616 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.604390276Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-16T14:32:42.616 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.607396489Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-16T14:32:42.616 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.608027512Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-16T14:32:42.616 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.610758102Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.611289603Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.613487887Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.61635159Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.616797901Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.6205868Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.623454446Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.626248409Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.626875648Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.629708738Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-16T14:32:42.861 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.632391409Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.635068736Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.635596009Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.638244147Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.640803192Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.643522069Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.643937683Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.646760634Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.64932442Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.652000818Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.652612952Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.655136716Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.657783136Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.660583625Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-16T14:32:42.862 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.661012921Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.663706093Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.664243619Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.666300835Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.667182802Z level=info msg="Executing migration" id="create data_source table" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.667629462Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.668581091Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.669228221Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.669983125Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.6704071Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.671413548Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.672207004Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.67309876Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.673817411Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-16T14:32:42.863 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.674196065Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.675150271Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.675806195Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.676692924Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.67717681Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.678051258Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.67901508Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.679762805Z level=info msg="Executing migration" id="Add uid column" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.680319589Z level=info msg="Executing migration" id="Update uid value" 2024-03-16T14:32:42.865 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.68114215Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.681570203Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.682038878Z level=info msg="Executing migration" id="create api_key table" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.682878547Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.683290529Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.684241355Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.685248442Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.685963389Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.686897588Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.687652213Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.688833501Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.689666481Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.690634047Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.691531959Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.692425566Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-16T14:32:42.866 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.69292945Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.69324834Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.693504025Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.694086784Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.694661596Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.694981973Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.695584923Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.696219034Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.698416561Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.698919036Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.701177277Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.7040329Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.706919264Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.709798251Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-16T14:32:42.867 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.712793813Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.718177401Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.721180469Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.724069349Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.726796059Z level=info msg="Executing migration" id="create quota table v1" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.72980587Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.732740534Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.733103907Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.735616323Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.738428536Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.741257295Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.741599692Z level=info msg="Executing migration" id="create session table" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.744216108Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.746687809Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.74698507Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-16T14:32:42.868 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.74741148Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.750176704Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.752844029Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.753163156Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.753449379Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.753764776Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.754173651Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.75443859Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.757014413Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.759577464Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.760216083Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.763123321Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.765508184Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.765966891Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.768880257Z level=info msg="Executing migration" id="create alert table v1" 2024-03-16T14:32:42.869 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.771702321Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.774479076Z level=info msg="Executing migration" id="add index alert state" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.777210599Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.77770041Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.778092055Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.778513627Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.778926595Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.780411343Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.78082087Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.781248568Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.781538306Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.78190717Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.782323455Z level=info msg="Executing migration" id="Add column is_default" 2024-03-16T14:32:42.870 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.78296579Z level=info msg="Executing migration" id="Add column frequency" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.783580979Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.784227895Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.784852886Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.785263902Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.785517338Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.785811656Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.786181343Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.78659766Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.786974961Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.787325801Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.787733474Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.788378964Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.789059177Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-16T14:32:42.871 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.789365158Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.78978679Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.790180287Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.790811271Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.791103685Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.791485438Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.79189773Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.792197092Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.792629466Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.793092144Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.793518026Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.793956272Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.7951231Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.797943325Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.800760809Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-16T14:32:42.872 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.801476047Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.804283244Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.807039995Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.809670845Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.8105128Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.811491029Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.813252647Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.814123116Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.815017703Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.815823716Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.816696637Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.817570709Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.818423139Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.819264559Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.820116341Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-16T14:32:42.873 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.82097171Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.821942011Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.822832039Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.82369615Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.82457156Z level=info msg="Executing migration" id="Move region to single row" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.825328882Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.826319418Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.827119146Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.828106928Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.828845593Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.829817482Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.830817745Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.831595136Z level=info msg="Executing migration" id="create test_data table" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.832369957Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-16T14:32:42.874 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.832798967Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.833765567Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.834527927Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.835516215Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.836267213Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.837067347Z level=info msg="Executing migration" id="create team table" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.837501405Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.837967704Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.838746649Z level=info msg="Executing migration" id="create team member table" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.839764967Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.840611177Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.841361826Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.842359154Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.843202751Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-16T14:32:42.875 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.844024527Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.84503715Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.8457515Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.846616016Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.847635875Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.848312847Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.849336499Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.850038459Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.850477476Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.851312053Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.852160085Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.853019574Z level=info msg="Executing migration" id="create tag table" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.853436836Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.856331786Z level=info msg="Executing migration" id="create login attempt table" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.859085226Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-16T14:32:42.876 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.861755578Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-16T14:32:42.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:42 smithi026 ceph-mon[25218]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:42.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:42 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:42.877 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:42 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:42.877 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:42 smithi026 ceph-mon[38287]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:42.877 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:42 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:42.877 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:42 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:43.111 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.864378157Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-16T14:32:43.111 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.867293357Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-16T14:32:43.111 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.870088182Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-16T14:32:43.111 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.872756562Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.873092788Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.876067461Z level=info msg="Executing migration" id="create user auth table" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.876476795Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.878783566Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.881475593Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.882409874Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.885232376Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.887997094Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.89088481Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.891369253Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.894117835Z level=info msg="Executing migration" id="create server_lock table" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.896727753Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.899609109Z level=info msg="Executing migration" id="create user auth token table" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.900081516Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-16T14:32:43.112 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.902661765Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.905391618Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.907884518Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.908694428Z level=info msg="Executing migration" id="create cache_data table" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.91136285Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.913890797Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.916771935Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.91725203Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.919674346Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.919980204Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.922344405Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-16T14:32:43.114 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.924882013Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.925322041Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.926169562Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.926600736Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.927032602Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.927849262Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.928334499Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.929173844Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.929995346Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.930983217Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.931661946Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.932106864Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.932896121Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.933206859Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-16T14:32:43.115 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.933790436Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.934475929Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.934919613Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.935360095Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.936191569Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.937165455Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.937919451Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.941397266Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.94479525Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.94563867Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.946683223Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.947694761Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.948579651Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.949391811Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-16T14:32:43.116 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.950411425Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.95091398Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.951767944Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.95265086Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.953491817Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.954333793Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.955193441Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.956098426Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.956946898Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.957787473Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.960643935Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.963332161Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.965982332Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-16T14:32:43.117 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.966309442Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.968986314Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.97178097Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.974423006Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.97474602Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.977707951Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.980634228Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.983349088Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.984216548Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.987109448Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.989855943Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.992754627Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.993648548Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.996370009Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-16T14:32:43.118 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:42.999205179Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.001790135Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.002702991Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.005337258Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.00795409Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.010514364Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.010996212Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.013742947Z level=info msg="Executing migration" id="create alert_image table" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.016224932Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.018849286Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.019253139Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.021985704Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.02470622Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.024807775Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-16T14:32:43.119 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.027313584Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.027887724Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.030712995Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.030943977Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.0310574Z level=info msg="alerts found to migrate" alerts=0 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.033308973Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.034029928Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.034532032Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.035400729Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.03615241Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.036953842Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.037265766Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.038052909Z level=info msg="Executing migration" id="create data_keys table" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.038539196Z level=info msg="Executing migration" id="create secrets table" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.039005721Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-16T14:32:43.120 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.043400668Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.04447982Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.047380561Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.051738316Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.056064454Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.056507542Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.059510839Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.062595471Z level=info msg="Executing migration" id="create permission table" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.065319381Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.065817613Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.068803109Z level=info msg="Executing migration" id="create role table" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.071626917Z level=info msg="Executing migration" id="add column display_name" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.074542026Z level=info msg="Executing migration" id="add column group_name" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.075517747Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-16T14:32:43.121 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.078378464Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.081340864Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.083829801Z level=info msg="Executing migration" id="create team role table" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.084296839Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.087143589Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.089709844Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.092418099Z level=info msg="Executing migration" id="create user role table" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.095177852Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.098018029Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.100705019Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.103596237Z level=info msg="Executing migration" id="create builtin role table" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.104090008Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.107032028Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.109616692Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-16T14:32:43.122 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.111107416Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-16T14:32:43.123 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.11241802Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-16T14:32:43.291 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-16T14:32:43.487 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.115127748Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-16T14:32:43.487 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.115575277Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.116033056Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.116430737Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.116891741Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.117888427Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.118321953Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.118783965Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.119082068Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.119482667Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.119967577Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.120328673Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.120623077Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.121012554Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-16T14:32:43.488 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.12143421Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-16T14:32:43.489 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.121936306Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-16T14:32:43.489 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.122933302Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-16T14:32:43.489 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.123253288Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-16T14:32:43.489 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.123674681Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.12412941Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.124576663Z level=info msg="Executing migration" id="add correlation config column" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.125584251Z level=info msg="Executing migration" id="create entity_events table" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.126046087Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.128624206Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.128716623Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.129116609Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.129208912Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.129466641Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.129860022Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-16T14:32:43.490 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.130266679Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.130698391Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.131127777Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.131516798Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.131939513Z level=info msg="Executing migration" id="Drop public config table" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.132474453Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.132915371Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.133310902Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.133767756Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.136499448Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.139683561Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.140722899Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.141777576Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.142103204Z level=info msg="Executing migration" id="add share column" 2024-03-16T14:32:43.491 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.14318755Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.143650636Z level=info msg="Executing migration" id="create file table" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.144121986Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.144936758Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.145472307Z level=info msg="Executing migration" id="create file_meta table" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.145879619Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.146727443Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.147696879Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.148468742Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.148811881Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.149659522Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.150706854Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.15168083Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.152327321Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-16T14:32:43.492 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.153187194Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.154036649Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.15500212Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.156242305Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.157363919Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.158200825Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.16816614Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.168647567Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.169530359Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.172943799Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.173779005Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.174181075Z level=info msg="Executing migration" id="create folder table" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.17499503Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.175501995Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-16T14:32:43.493 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.176047178Z level=info msg="Executing migration" id="Update folder title length" 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.176803905Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=migrator t=2024-03-16T14:32:43.177367938Z level=info msg="migrations completed" performed=485 skipped=0 duration=655.180537ms 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=sqlstore t=2024-03-16T14:32:43.177813365Z level=info msg="Created default organization" 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=plugin.loader t=2024-03-16T14:32:43.23258166Z level=info msg="Plugin registered" pluginID=input 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=plugin.loader t=2024-03-16T14:32:43.260059866Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=plugin.loader t=2024-03-16T14:32:43.260089846Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=secrets t=2024-03-16T14:32:43.260123856Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=query_data t=2024-03-16T14:32:43.26221519Z level=info msg="Query Service initialization" 2024-03-16T14:32:43.494 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=live.push_http t=2024-03-16T14:32:43.264654554Z level=info msg="Live Push Gateway initialization" 2024-03-16T14:32:43.541 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:43.565 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[25218]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:43.565 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:43.565 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:43.565 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[25218]: Deploying daemon node-exporter.a on smithi026 2024-03-16T14:32:43.898 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-mon[30603]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:43.899 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:43.899 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:43.899 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-mon[30603]: Deploying daemon node-exporter.a on smithi026 2024-03-16T14:32:43.950 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[38287]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:43.950 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:43.950 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:43.950 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:43 smithi026 ceph-mon[38287]: Deploying daemon node-exporter.a on smithi026 2024-03-16T14:32:44.237 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=infra.usagestats.collector t=2024-03-16T14:32:43.897336384Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-16T14:32:44.237 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=provisioning.datasources t=2024-03-16T14:32:43.898692855Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-16T14:32:44.237 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=provisioning.datasources t=2024-03-16T14:32:43.909950501Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-16T14:32:44.237 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=provisioning.alerting t=2024-03-16T14:32:43.921168448Z level=info msg="starting to provision alerting" 2024-03-16T14:32:44.237 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=provisioning.alerting t=2024-03-16T14:32:43.921188915Z level=info msg="finished to provision alerting" 2024-03-16T14:32:44.237 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=ngalert.state.manager t=2024-03-16T14:32:43.92133767Z level=info msg="Warming state cache for startup" 2024-03-16T14:32:44.238 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=grafanaStorageLogger t=2024-03-16T14:32:43.921416603Z level=info msg="storage starting" 2024-03-16T14:32:44.238 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=http.server t=2024-03-16T14:32:43.923159914Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-16T14:32:44.238 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=ngalert.state.manager t=2024-03-16T14:32:43.926022949Z level=info msg="State cache has been initialized" states=0 duration=4.684557ms 2024-03-16T14:32:44.238 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=ticker t=2024-03-16T14:32:43.926076139Z level=info msg=starting first_tick=2024-03-16T14:32:50Z 2024-03-16T14:32:44.238 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=ngalert.multiorg.alertmanager t=2024-03-16T14:32:43.926086768Z level=info msg="starting MultiOrg Alertmanager" 2024-03-16T14:32:44.238 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=sqlstore.transactions t=2024-03-16T14:32:43.932260379Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-16T14:32:44.238 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:32:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a[70841]: logger=sqlstore.transactions t=2024-03-16T14:32:43.942489851Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-16T14:32:44.986 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:44 smithi151 ceph-mon[30603]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:45.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:44 smithi026 ceph-mon[38287]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:45.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:44 smithi026 ceph-mon[25218]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:45.795 INFO:teuthology.orchestra.run.smithi026.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-16T14:32:45.795 INFO:teuthology.orchestra.run.smithi026.stdout:alertmanager ?:9093,9094 0/1 - 84s smithi026=a;count:1 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:grafana ?:3000 0/1 - 80s smithi151=a;count:1 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:iscsi.datapool ?:5000 1/1 56s ago 94s smithi151=iscsi.a;count:1 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:mgr 2/2 56s ago 5m smithi026=y;smithi151=x;count:2 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:mon 3/3 56s ago 5m smithi026:172.21.15.26=a;smithi026:[v2:172.21.15.26:3301,v1:172.21.15.26:6790]=c;smithi151:172.21.15.151=b;count:3 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:node-exporter ?:9100 0/2 - 86s smithi026=a;smithi151=b;count:2 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:osd 8 56s ago - 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:prometheus ?:9095 1/1 56s ago 91s smithi151=a;count:1 2024-03-16T14:32:45.796 INFO:teuthology.orchestra.run.smithi026.stdout:rgw.foo.a ?:80 1/1 31s ago 102s smithi026=foo.a;count:1 2024-03-16T14:32:46.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:46 smithi151 ceph-mon[30603]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:46.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:46 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:46.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:46 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:46.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:46 smithi026 ceph-mon[38287]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:46.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:46 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:46.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:46 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:46.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:46 smithi026 ceph-mon[25218]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:46.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:46 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:46.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:46 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:32:47.114 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:32:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:32:47.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-16T14:32:47.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:47 smithi151 ceph-mon[30603]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:47.496 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:47.504 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:47 smithi026 ceph-mon[25218]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:47.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:47 smithi026 ceph-mon[38287]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:47.505 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:47 smithi026 bash[82848]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-16T14:32:47.814 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:47 smithi026 bash[82848]: Writing manifest to image destination 2024-03-16T14:32:48.110 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:47 smithi026 podman[82848]: 2024-03-16 14:32:47.766823107 +0000 UTC m=+2.681673484 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-16T14:32:48.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:48 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:48.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:48 smithi151 ceph-mon[30603]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:48.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:48 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:48.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:48 smithi026 ceph-mon[38287]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:48.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:48 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:48.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:48 smithi026 ceph-mon[25218]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:48.887 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:48 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:32:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:32:49.137 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:48 smithi026 podman[82848]: 2024-03-16 14:32:48.922011449 +0000 UTC m=+3.836861805 container create 3ef24dbba39fb474c538e4aa562c830134b41067fb6b4eab89deb3c364501d21 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-16T14:32:49.137 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 podman[82848]: 2024-03-16 14:32:49.132634039 +0000 UTC m=+4.047484409 container init 3ef24dbba39fb474c538e4aa562c830134b41067fb6b4eab89deb3c364501d21 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-16T14:32:49.137 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 podman[82848]: 2024-03-16 14:32:49.135902536 +0000 UTC m=+4.050752894 container start 3ef24dbba39fb474c538e4aa562c830134b41067fb6b4eab89deb3c364501d21 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-16T14:32:49.410 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.140Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.140Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.140Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.140Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=arp 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-16T14:32:49.411 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=edac 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-16T14:32:49.412 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=os 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-16T14:32:49.413 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=stat 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=time 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=uname 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.141Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.142Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a[83062]: ts=2024-03-16T14:32:49.142Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 bash[82848]: 3ef24dbba39fb474c538e4aa562c830134b41067fb6b4eab89deb3c364501d21 2024-03-16T14:32:49.414 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:32:49 smithi026 systemd[1]: Started Ceph node-exporter.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:32:50.041 INFO:teuthology.orchestra.run.smithi026.stdout:HOST ADDR LABELS STATUS 2024-03-16T14:32:50.041 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 172.21.15.26 2024-03-16T14:32:50.041 INFO:teuthology.orchestra.run.smithi026.stdout:smithi151 172.21.15.151 2024-03-16T14:32:50.042 INFO:teuthology.orchestra.run.smithi026.stdout:2 hosts in cluster 2024-03-16T14:32:50.555 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:50 smithi151 ceph-mon[30603]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:50.555 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:50 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.555 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:50 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.555 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:50 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.555 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:50 smithi151 ceph-mon[30603]: Deploying daemon node-exporter.b on smithi151 2024-03-16T14:32:50.555 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:50 smithi151 systemd[1]: Starting Ceph node-exporter.b for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[38287]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[38287]: Deploying daemon node-exporter.b on smithi151 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[25218]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:50.756 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:50 smithi026 ceph-mon[25218]: Deploying daemon node-exporter.b on smithi151 2024-03-16T14:32:50.913 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:50 smithi151 bash[71106]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-16T14:32:50.913 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:50 smithi151 bash[71106]: Getting image source signatures 2024-03-16T14:32:50.913 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:50 smithi151 bash[71106]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-16T14:32:50.913 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:50 smithi151 bash[71106]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-16T14:32:50.913 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:50 smithi151 bash[71106]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-16T14:32:50.987 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-16T14:32:51.239 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:51.554 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:51 smithi026 ceph-mon[25218]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:51.554 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:51 smithi026 ceph-mon[38287]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:51.736 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:51 smithi151 ceph-mon[30603]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:52.389 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:52 smithi026 ceph-mon[38287]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:52.390 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:52 smithi026 ceph-mon[25218]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:52.685 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:52 smithi151 bash[71106]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-16T14:32:52.685 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:52 smithi151 bash[71106]: Writing manifest to image destination 2024-03-16T14:32:52.686 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:52 smithi151 ceph-mon[30603]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:52.987 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:52 smithi151 podman[71106]: 2024-03-16 14:32:52.659875582 +0000 UTC m=+2.199165175 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-16T14:32:52.987 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:52 smithi151 podman[71106]: 2024-03-16 14:32:52.766934597 +0000 UTC m=+2.306224189 container create a06460fc71da51228755ec02c4137ca6e99f039c1f583633a6cd887a16418b56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-16T14:32:53.149 INFO:teuthology.orchestra.run.smithi026.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-16T14:32:53.149 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760020480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-16T14:32:53.149 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme1n1 ssd Linux_fce080e6f8ef851636ca 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.149 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme2n1 ssd Linux_0c6f6191e7343f2dbecc 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.150 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme3n1 ssd Linux_05203bf964821021fac4 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.150 INFO:teuthology.orchestra.run.smithi026.stdout:smithi026 /dev/nvme4n1 ssd Linux_c0d34abe40ec614719cb 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.150 INFO:teuthology.orchestra.run.smithi026.stdout:smithi151 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JJ400BGN 372G No 118s ago Has a FileSystem, LVM detected 2024-03-16T14:32:53.150 INFO:teuthology.orchestra.run.smithi026.stdout:smithi151 /dev/nvme1n1 ssd Linux_feac4ad875fb621cce0c 89.4G No 118s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.150 INFO:teuthology.orchestra.run.smithi026.stdout:smithi151 /dev/nvme2n1 ssd Linux_7d915526d95c7031a872 89.4G No 118s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.150 INFO:teuthology.orchestra.run.smithi026.stdout:smithi151 /dev/nvme3n1 ssd Linux_142f0dd0ca95b7716786 89.4G No 118s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.150 INFO:teuthology.orchestra.run.smithi026.stdout:smithi151 /dev/nvme4n1 ssd Linux_6ad22d7cdd05cb024c38 89.4G No 118s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-16T14:32:53.488 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 podman[71106]: 2024-03-16 14:32:53.010792354 +0000 UTC m=+2.550081944 container init a06460fc71da51228755ec02c4137ca6e99f039c1f583633a6cd887a16418b56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-16T14:32:53.488 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 podman[71106]: 2024-03-16 14:32:53.013653264 +0000 UTC m=+2.552942867 container start a06460fc71da51228755ec02c4137ca6e99f039c1f583633a6cd887a16418b56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-16T14:32:53.488 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.017Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-16T14:32:53.488 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.017Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-16T14:32:53.488 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-16T14:32:53.488 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=node_exporter.go:117 level=info collector=arp 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.018Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-16T14:32:53.489 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-16T14:32:53.490 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-16T14:32:53.490 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-16T14:32:53.491 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=edac 2024-03-16T14:32:53.491 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-16T14:32:53.491 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-16T14:32:53.491 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-16T14:32:53.491 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-16T14:32:53.491 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-16T14:32:53.492 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=os 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-16T14:32:53.493 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=stat 2024-03-16T14:32:53.494 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-16T14:32:53.494 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-16T14:32:53.494 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-16T14:32:53.494 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=time 2024-03-16T14:32:53.494 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-16T14:32:53.494 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=uname 2024-03-16T14:32:53.494 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-16T14:32:53.495 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-16T14:32:53.495 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-16T14:32:53.495 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.019Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-16T14:32:53.495 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b[71156]: ts=2024-03-16T14:32:53.020Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-16T14:32:53.495 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 bash[71106]: a06460fc71da51228755ec02c4137ca6e99f039c1f583633a6cd887a16418b56 2024-03-16T14:32:53.495 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:32:53 smithi151 systemd[1]: Started Ceph node-exporter.b for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:32:53.843 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-16T14:32:54.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:54.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:54.460 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.461 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:54 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:54.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:54 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-16T14:32:54.633 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:56.436 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:56 smithi151 ceph-mon[30603]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:56.436 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:56 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:56.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:56 smithi026 ceph-mon[38287]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:56.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:56 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:56.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:56 smithi026 ceph-mon[25218]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:56.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:56 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:56.659 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: alertmanager 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: alertmanager 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: count: 1 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi026=a 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:31:21.733256Z' 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: ports: 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: - 9093 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: - 9094 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: running: 0 2024-03-16T14:32:56.660 INFO:teuthology.orchestra.run.smithi026.stdout: size: 1 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout:events: 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout:- 2024-03-16T14:32:27.677535Z service:alertmanager [INFO] "service was created" 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: grafana 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: grafana 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: count: 1 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi151=a 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout:spec: 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: anonymous_access: true 2024-03-16T14:32:56.661 INFO:teuthology.orchestra.run.smithi026.stdout: protocol: https 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:31:24.801100Z' 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: ports: 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: - 3000 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: running: 0 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: size: 1 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:events: 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:- 2024-03-16T14:32:42.570778Z service:grafana [INFO] "service was created" 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: iscsi 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:service_id: datapool 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: iscsi.datapool 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: count: 1 2024-03-16T14:32:56.662 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi151=iscsi.a 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout:spec: 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: api_password: admin 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: api_port: 5000 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: api_user: admin 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: pool: datapool 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: trusted_ip_list: 172.21.15.151 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:31:11.275435Z' 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: last_refresh: '2024-03-16T14:31:49.380560Z' 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: ports: 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: - 5000 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: running: 1 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout: size: 1 2024-03-16T14:32:56.663 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: mgr 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: mgr 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: count: 2 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi026=y 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi151=x 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:27:25.155920Z' 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: last_refresh: '2024-03-16T14:31:49.380156Z' 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: running: 2 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout: size: 2 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.664 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: mon 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: mon 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: count: 3 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi026:172.21.15.26=a 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi026:[v2:172.21.15.26:3301,v1:172.21.15.26:6790]=c 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi151:172.21.15.151=b 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:27:00.675755Z' 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: last_refresh: '2024-03-16T14:31:49.380048Z' 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: running: 3 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout: size: 3 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: node-exporter 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: node-exporter 2024-03-16T14:32:56.665 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: count: 2 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi026=a 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi151=b 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:31:18.838088Z' 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: ports: 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: - 9100 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: running: 0 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout: size: 2 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout:events: 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout:- 2024-03-16T14:32:53.186533Z service:node-exporter [INFO] "service was created" 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.666 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: osd 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: osd 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout:unmanaged: true 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout:spec: 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: filter_logic: AND 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: objectstore: bluestore 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: container_image_id: 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: last_refresh: '2024-03-16T14:31:49.380241Z' 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: running: 8 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout: size: 8 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: prometheus 2024-03-16T14:32:56.667 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: prometheus 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: count: 1 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi151=a 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:31:14.186550Z' 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: last_refresh: '2024-03-16T14:31:49.380641Z' 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: ports: 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: - 9095 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: running: 1 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout: size: 1 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout:events: 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout:- 2024-03-16T14:32:53.192827Z service:prometheus [INFO] "service was created" 2024-03-16T14:32:56.668 INFO:teuthology.orchestra.run.smithi026.stdout:--- 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout:service_type: rgw 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout:service_id: foo.a 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout:service_name: rgw.foo.a 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout:placement: 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: count: 1 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: hosts: 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: - smithi026=foo.a 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout:status: 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: created: '2024-03-16T14:31:03.283901Z' 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: last_refresh: '2024-03-16T14:32:13.873810Z' 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: ports: 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: - 80 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: running: 1 2024-03-16T14:32:56.669 INFO:teuthology.orchestra.run.smithi026.stdout: size: 1 2024-03-16T14:32:57.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:32:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:32:58.089 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-16T14:32:58.415 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[38287]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-16T14:32:58.415 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[38287]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:58.415 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[38287]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:58.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:58.416 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[25218]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-16T14:32:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[25218]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[25218]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:58.416 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:58 smithi151 ceph-mon[30603]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-16T14:32:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:58 smithi151 ceph-mon[30603]: from='client.14532 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-16T14:32:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:58 smithi151 ceph-mon[30603]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:32:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:58 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:58.487 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:58 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:58.742 INFO:teuthology.orchestra.run.smithi026.stderr:Inferring config /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/mon.a/config 2024-03-16T14:32:58.755 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:32:58 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:32:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:32:59.852 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: Reconfiguring daemon alertmanager.a on smithi026 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[38287]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:32:59.853 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:32:59.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:32:59.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-16T14:32:59.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: Reconfiguring daemon alertmanager.a on smithi026 2024-03-16T14:32:59.854 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:32:59 smithi026 ceph-mon[25218]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: from='mgr.14574 ' entity='mgr.y' 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: Reconfiguring daemon alertmanager.a on smithi026 2024-03-16T14:33:00.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:32:59 smithi151 ceph-mon[30603]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:33:01.005 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:00 smithi026 ceph-mon[38287]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:33:01.005 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:00 smithi026 ceph-mon[25218]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:33:01.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:00 smithi151 ceph-mon[30603]: from='mgr.14574 172.21.15.26:0/4268324578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-16T14:33:01.911 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:01 smithi026 ceph-mon[25218]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:33:01.911 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:01 smithi026 ceph-mon[38287]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:33:02.063 INFO:teuthology.orchestra.run.smithi026.stdout:osd 8 4s ago - 2024-03-16T14:33:02.236 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:01 smithi151 ceph-mon[30603]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-16T14:33:02.976 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:02 smithi026 ceph-mon[38287]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:33:02.977 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:02 smithi026 ceph-mon[25218]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:33:03.236 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:02 smithi151 ceph-mon[30603]: from='client.24659 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-16T14:33:03.371 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-16T14:33:03.379 INFO:tasks.cephadm:Teardown begin 2024-03-16T14:33:03.380 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T14:33:03.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T14:33:03.446 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-16T14:33:03.446 DEBUG:teuthology.orchestra.run.smithi026:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T14:33:03.464 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-16T14:33:03.503 INFO:tasks.cephadm:Stopping all daemons... 2024-03-16T14:33:03.503 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-16T14:33:03.503 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a 2024-03-16T14:33:03.755 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:03 smithi026 systemd[1]: Stopping Ceph mon.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:04.152 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:03 smithi026 systemd[1]: Stopping Ceph alertmanager.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:04.152 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:03 smithi026 ceph-mon[25218]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:33:04.152 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:03 smithi026 ceph-mon[38287]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:33:04.236 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:03 smithi151 ceph-mon[30603]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-16T14:33:04.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:04 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a[25214]: 2024-03-16T14:33:04.149+0000 7f10b4521700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:04.506 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:04 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a[25214]: 2024-03-16T14:33:04.149+0000 7f10b4521700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-16T14:33:04.506 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:04 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[81692]: ts=2024-03-16T14:33:04.308Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-16T14:33:04.506 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:04 smithi026 podman[84402]: 2024-03-16 14:33:04.328987122 +0000 UTC m=+0.124583445 container died 8a70f394d4fab3b617e0acc11a25f4f10452de7739385693edfe6a3649f79a19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:33:04.880 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:04 smithi026 podman[84402]: 2024-03-16 14:33:04.685979648 +0000 UTC m=+0.481575979 container cleanup 8a70f394d4fab3b617e0acc11a25f4f10452de7739385693edfe6a3649f79a19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:33:04.880 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:04 smithi026 bash[84402]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a 2024-03-16T14:33:05.242 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:04 smithi026 podman[84416]: 2024-03-16 14:33:04.878578767 +0000 UTC m=+0.546938990 container remove 8a70f394d4fab3b617e0acc11a25f4f10452de7739385693edfe6a3649f79a19 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:33:05.242 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:04 smithi026 podman[84416]: 2024-03-16 14:33:04.954150386 +0000 UTC m=+0.622510571 volume remove 82afb6ffc688e9addaddb43a48bfcb38633f6adec779e4aac5687c896468f545 2024-03-16T14:33:05.242 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:04 smithi026 podman[84375]: 2024-03-16 14:33:04.982417851 +0000 UTC m=+0.930749803 container died 3b7c51472ad641330f72f717db07ee1433f2e67b0ad73815d12547dcdd6db928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-16T14:33:05.504 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:05 smithi026 podman[84375]: 2024-03-16 14:33:05.431135271 +0000 UTC m=+1.379467232 container cleanup 3b7c51472ad641330f72f717db07ee1433f2e67b0ad73815d12547dcdd6db928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-16T14:33:05.505 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:05 smithi026 bash[84375]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a 2024-03-16T14:33:05.968 INFO:journalctl@ceph.mon.a.smithi026.stdout:Mar 16 14:33:05 smithi026 podman[84427]: 2024-03-16 14:33:05.682389992 +0000 UTC m=+1.329963527 container remove 3b7c51472ad641330f72f717db07ee1433f2e67b0ad73815d12547dcdd6db928 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:05.968 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:05 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-16T14:33:05.968 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:05 smithi026 systemd[1]: Stopped Ceph alertmanager.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:33:05.968 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:05 smithi026 systemd[1]: Starting Ceph alertmanager.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:05.968 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:05 smithi026 podman[84545]: 2024-03-16 14:33:05.829908984 +0000 UTC m=+0.017976525 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-16T14:33:05.968 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:05 smithi026 podman[84545]: 2024-03-16 14:33:05.966939828 +0000 UTC m=+0.155007354 volume create cb11a67b24c65675074dd856be1f67c2ba8d17d4702b25b6bd207ae56a51c9d7 2024-03-16T14:33:06.242 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 podman[84545]: 2024-03-16 14:33:06.059221182 +0000 UTC m=+0.247288708 container create ec5ad52188926b68daafe1b225d18cfbc3b48b648184ad3eee6683769f06a7ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:33:06.488 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.a.service' 2024-03-16T14:33:06.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:06.544 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-16T14:33:06.545 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-16T14:33:06.545 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.c 2024-03-16T14:33:06.595 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 podman[84545]: 2024-03-16 14:33:06.33682956 +0000 UTC m=+0.524897087 container init ec5ad52188926b68daafe1b225d18cfbc3b48b648184ad3eee6683769f06a7ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:33:06.595 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 podman[84545]: 2024-03-16 14:33:06.339427344 +0000 UTC m=+0.527494873 container start ec5ad52188926b68daafe1b225d18cfbc3b48b648184ad3eee6683769f06a7ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.351Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.351Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.352Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.26 port=9094 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.353Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.376Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.377Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.379Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:06.379Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 bash[84545]: ec5ad52188926b68daafe1b225d18cfbc3b48b648184ad3eee6683769f06a7ab 2024-03-16T14:33:06.596 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:06 smithi026 systemd[1]: Started Ceph alertmanager.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:33:06.858 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:06 smithi026 systemd[1]: Stopping Ceph mon.c for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c[38283]: 2024-03-16T14:33:06.855+0000 7f01b5fb4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c[38283]: 2024-03-16T14:33:06.855+0000 7f01b5fb4700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:06 smithi026 podman[84642]: 2024-03-16 14:33:06.927804874 +0000 UTC m=+0.192795077 container died 12f7db96a36003427ad7bb99f2dc6551ac268853a79e022f63007196132ecfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:33:06] ENGINE Bus STOPPING 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:33:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:33:06] ENGINE Bus STOPPED 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:33:06] ENGINE Bus STARTING 2024-03-16T14:33:07.217 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:33:06] ENGINE Serving on http://:::9283 2024-03-16T14:33:07.218 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:06 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: [16/Mar/2024:14:33:06] ENGINE Bus STARTED 2024-03-16T14:33:07.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:33:06 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug there is no tcmu-runner data available 2024-03-16T14:33:07.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:07 smithi026 podman[84642]: 2024-03-16 14:33:07.373270386 +0000 UTC m=+0.638260588 container cleanup 12f7db96a36003427ad7bb99f2dc6551ac268853a79e022f63007196132ecfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:07.505 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:07 smithi026 bash[84642]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c 2024-03-16T14:33:07.798 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.c.service' 2024-03-16T14:33:07.821 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:07 smithi026 podman[84667]: 2024-03-16 14:33:07.549473586 +0000 UTC m=+0.615760296 container remove 12f7db96a36003427ad7bb99f2dc6551ac268853a79e022f63007196132ecfea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-c, ceph=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-16T14:33:07.821 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:07 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-16T14:33:07.821 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:07 smithi026 systemd[1]: Stopped Ceph mon.c for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:33:07.821 INFO:journalctl@ceph.mon.c.smithi026.stdout:Mar 16 14:33:07 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.c.service: Consumed 4.599s CPU time. 2024-03-16T14:33:07.834 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:07.835 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-16T14:33:07.835 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-16T14:33:07.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.b 2024-03-16T14:33:08.237 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:07 smithi151 systemd[1]: Stopping Ceph mon.b for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:08.608 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:08.354Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000314272s 2024-03-16T14:33:09.004 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:08 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: ::ffff:172.21.15.151 - - [16/Mar/2024:14:33:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-16T14:33:09.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-b[30587]: 2024-03-16T14:33:09.501+0000 7fba52c49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:09.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-b[30587]: 2024-03-16T14:33:09.501+0000 7fba52c49700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-16T14:33:09.987 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:09 smithi151 podman[71943]: 2024-03-16 14:33:09.584847112 +0000 UTC m=+1.546000042 container died 10f194751dd69b2a837123a3eb30f2a998741a414020ea410c5f7fd4ca9de02e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-16T14:33:10.338 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:10 smithi151 podman[71943]: 2024-03-16 14:33:10.077044372 +0000 UTC m=+2.038197314 container cleanup 10f194751dd69b2a837123a3eb30f2a998741a414020ea410c5f7fd4ca9de02e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-03-16T14:33:10.338 INFO:journalctl@ceph.mon.b.smithi151.stdout:Mar 16 14:33:10 smithi151 bash[71943]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mon-b 2024-03-16T14:33:10.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mon.b.service' 2024-03-16T14:33:10.570 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:10.570 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-16T14:33:10.571 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-16T14:33:10.571 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y 2024-03-16T14:33:10.887 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:10 smithi026 systemd[1]: Stopping Ceph mgr.y for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:11.156 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:10 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:33:10.885+0000 7f7210a44700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:11.156 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:10 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y[25450]: 2024-03-16T14:33:10.885+0000 7f7210a44700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-16T14:33:11.156 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:10 smithi026 podman[84773]: 2024-03-16 14:33:10.916566322 +0000 UTC m=+0.147742655 container died cfd474663860976b1402d5f59900e857957aa5fbe47f14f0a43396bfe538e8dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-16T14:33:11.505 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:11 smithi026 podman[84773]: 2024-03-16 14:33:11.3363652 +0000 UTC m=+0.567541574 container cleanup cfd474663860976b1402d5f59900e857957aa5fbe47f14f0a43396bfe538e8dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-16T14:33:11.505 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:11 smithi026 bash[84773]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y 2024-03-16T14:33:11.764 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-16T14:33:11.787 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:11 smithi026 podman[84787]: 2024-03-16 14:33:11.529143684 +0000 UTC m=+0.613055471 container remove cfd474663860976b1402d5f59900e857957aa5fbe47f14f0a43396bfe538e8dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5ea4d9cfac4d9f2893126a48d28cca7da69243c1, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-16T14:33:11.787 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:11 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y.service: Deactivated successfully. 2024-03-16T14:33:11.787 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:11 smithi026 systemd[1]: Stopped Ceph mgr.y for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:33:11.787 INFO:journalctl@ceph.mgr.y.smithi026.stdout:Mar 16 14:33:11 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.y.service: Consumed 56.872s CPU time. 2024-03-16T14:33:11.800 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:11.800 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-16T14:33:11.800 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-16T14:33:11.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.x 2024-03-16T14:33:12.122 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:33:11 smithi151 systemd[1]: Stopping Ceph mgr.x for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:12.433 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:33:12 smithi151 podman[72066]: 2024-03-16 14:33:12.12074977 +0000 UTC m=+0.139312524 container died 24f82a3b44e474fe32befd148cb679b4b6874383795cd1d91eeee29ff425b4c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:12.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:33:12 smithi151 podman[72066]: 2024-03-16 14:33:12.431576427 +0000 UTC m=+0.450139192 container cleanup 24f82a3b44e474fe32befd148cb679b4b6874383795cd1d91eeee29ff425b4c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:33:12.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:33:12 smithi151 bash[72066]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x 2024-03-16T14:33:12.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:33:12 smithi151 podman[72080]: 2024-03-16 14:33:12.616257914 +0000 UTC m=+0.496669382 container remove 24f82a3b44e474fe32befd148cb679b4b6874383795cd1d91eeee29ff425b4c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307) 2024-03-16T14:33:12.737 INFO:journalctl@ceph.mgr.x.smithi151.stdout:Mar 16 14:33:12 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-16T14:33:12.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-16T14:33:12.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:12.901 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-16T14:33:12.902 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-16T14:33:12.902 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.0 2024-03-16T14:33:13.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:12 smithi026 systemd[1]: Stopping Ceph osd.0 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:13.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:13 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0[42155]: 2024-03-16T14:33:13.189+0000 7ff33d2f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:13.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:13 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0[42155]: 2024-03-16T14:33:13.189+0000 7ff33d2f5700 -1 osd.0 71 *** Got signal Terminated *** 2024-03-16T14:33:13.193 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:13 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0[42155]: 2024-03-16T14:33:13.189+0000 7ff33d2f5700 -1 osd.0 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:33:16.755 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:33:16 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:33:16.356Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002544927s 2024-03-16T14:33:18.993 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:18 smithi026 podman[84895]: 2024-03-16 14:33:18.696724746 +0000 UTC m=+5.602530563 container died b7a6b3dfa22993c3675280b4943e0ccc6e3adbcb19deb45ef68ffc6a1efccb7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-16T14:33:19.254 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:19 smithi026 podman[84895]: 2024-03-16 14:33:19.165431549 +0000 UTC m=+6.071237408 container cleanup b7a6b3dfa22993c3675280b4943e0ccc6e3adbcb19deb45ef68ffc6a1efccb7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-16T14:33:19.255 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:19 smithi026 bash[84895]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0 2024-03-16T14:33:19.735 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:19 smithi026 podman[84924]: 2024-03-16 14:33:19.416617924 +0000 UTC m=+0.719787702 container remove b7a6b3dfa22993c3675280b4943e0ccc6e3adbcb19deb45ef68ffc6a1efccb7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-03-16T14:33:20.005 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:19 smithi026 podman[85014]: 2024-03-16 14:33:19.734172353 +0000 UTC m=+0.100953172 container create d69e6322b022ff79311476dc388ccf0e1a157d6eef3b00c238db5aac34eed196 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-16T14:33:20.005 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:19 smithi026 podman[85014]: 2024-03-16 14:33:19.651311487 +0000 UTC m=+0.018092316 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:33:20.005 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:19 smithi026 podman[85014]: 2024-03-16 14:33:19.978320957 +0000 UTC m=+0.345101775 container init d69e6322b022ff79311476dc388ccf0e1a157d6eef3b00c238db5aac34eed196 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-16T14:33:20.005 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:19 smithi026 podman[85014]: 2024-03-16 14:33:19.981521888 +0000 UTC m=+0.348302714 container start d69e6322b022ff79311476dc388ccf0e1a157d6eef3b00c238db5aac34eed196 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-16T14:33:20.505 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:20 smithi026 podman[85014]: 2024-03-16 14:33:20.061744679 +0000 UTC m=+0.428525509 container attach d69e6322b022ff79311476dc388ccf0e1a157d6eef3b00c238db5aac34eed196 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T14:33:20.505 INFO:journalctl@ceph.osd.0.smithi026.stdout:Mar 16 14:33:20 smithi026 podman[85014]: 2024-03-16 14:33:20.284060611 +0000 UTC m=+0.650841431 container died d69e6322b022ff79311476dc388ccf0e1a157d6eef3b00c238db5aac34eed196 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:21.018 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.0.service' 2024-03-16T14:33:21.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:21.062 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-16T14:33:21.063 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-16T14:33:21.063 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.1 2024-03-16T14:33:21.254 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:21 smithi026 systemd[1]: Stopping Ceph osd.1 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:21.754 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:21 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1[50054]: 2024-03-16T14:33:21.376+0000 7f463737a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:21.755 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:21 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1[50054]: 2024-03-16T14:33:21.376+0000 7f463737a700 -1 osd.1 71 *** Got signal Terminated *** 2024-03-16T14:33:21.755 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:21 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1[50054]: 2024-03-16T14:33:21.376+0000 7f463737a700 -1 osd.1 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:33:27.062 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:26 smithi026 podman[85120]: 2024-03-16 14:33:26.757723385 +0000 UTC m=+5.477719358 container died ff7440a76d74a584f2752998cf66720347a3bca45ea2a801d61d0091a36ebfac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:33:27.504 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:27 smithi026 podman[85120]: 2024-03-16 14:33:27.268247888 +0000 UTC m=+5.988243905 container cleanup ff7440a76d74a584f2752998cf66720347a3bca45ea2a801d61d0091a36ebfac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T14:33:27.504 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:27 smithi026 bash[85120]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1 2024-03-16T14:33:27.754 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:27 smithi026 podman[85137]: 2024-03-16 14:33:27.50295488 +0000 UTC m=+0.747098630 container remove ff7440a76d74a584f2752998cf66720347a3bca45ea2a801d61d0091a36ebfac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-16T14:33:28.190 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:27 smithi026 podman[85232]: 2024-03-16 14:33:27.84113558 +0000 UTC m=+0.018421638 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:33:28.190 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:27 smithi026 podman[85232]: 2024-03-16 14:33:27.945295539 +0000 UTC m=+0.122581609 container create ea65d1c6cf1317d675eb62cce691e120a6edf94e4488fdb9c755a096ff917c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:28.496 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:28 smithi026 podman[85232]: 2024-03-16 14:33:28.189157398 +0000 UTC m=+0.366443423 container init ea65d1c6cf1317d675eb62cce691e120a6edf94e4488fdb9c755a096ff917c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-16T14:33:28.496 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:28 smithi026 podman[85232]: 2024-03-16 14:33:28.192334574 +0000 UTC m=+0.369620608 container start ea65d1c6cf1317d675eb62cce691e120a6edf94e4488fdb9c755a096ff917c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-16T14:33:28.496 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:28 smithi026 podman[85232]: 2024-03-16 14:33:28.264477979 +0000 UTC m=+0.441764020 container attach ea65d1c6cf1317d675eb62cce691e120a6edf94e4488fdb9c755a096ff917c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T14:33:28.755 INFO:journalctl@ceph.osd.1.smithi026.stdout:Mar 16 14:33:28 smithi026 podman[85232]: 2024-03-16 14:33:28.495124076 +0000 UTC m=+0.672410104 container died ea65d1c6cf1317d675eb62cce691e120a6edf94e4488fdb9c755a096ff917c40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-16T14:33:29.037 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.1.service' 2024-03-16T14:33:29.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:29.079 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-16T14:33:29.079 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-16T14:33:29.079 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.2 2024-03-16T14:33:29.397 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:29 smithi026 systemd[1]: Stopping Ceph osd.2 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:29.754 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:29 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2[57988]: 2024-03-16T14:33:29.393+0000 7f030d797700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:29.755 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:29 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2[57988]: 2024-03-16T14:33:29.393+0000 7f030d797700 -1 osd.2 71 *** Got signal Terminated *** 2024-03-16T14:33:29.755 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:29 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2[57988]: 2024-03-16T14:33:29.393+0000 7f030d797700 -1 osd.2 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:33:35.198 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:34 smithi026 podman[85339]: 2024-03-16 14:33:34.918243408 +0000 UTC m=+5.620633287 container died 5b3fbc0a4cd78f1451c097e990fcd937ed308298d0b0bf99435eb47e4b9cdfd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T14:33:35.504 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:35 smithi026 podman[85339]: 2024-03-16 14:33:35.370038051 +0000 UTC m=+6.072427927 container cleanup 5b3fbc0a4cd78f1451c097e990fcd937ed308298d0b0bf99435eb47e4b9cdfd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:35.505 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:35 smithi026 bash[85339]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2 2024-03-16T14:33:35.989 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:35 smithi026 podman[85356]: 2024-03-16 14:33:35.637913348 +0000 UTC m=+0.716341241 container remove 5b3fbc0a4cd78f1451c097e990fcd937ed308298d0b0bf99435eb47e4b9cdfd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-16T14:33:36.255 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:35 smithi026 podman[85446]: 2024-03-16 14:33:35.890240013 +0000 UTC m=+0.017866419 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:33:36.255 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:36 smithi026 podman[85446]: 2024-03-16 14:33:36.005643523 +0000 UTC m=+0.133269920 container create 9891ea5c19e81dcece9932249c20db86ad5fedafdfc74da3c99651ff586872be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-16T14:33:36.255 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:36 smithi026 podman[85446]: 2024-03-16 14:33:36.232827957 +0000 UTC m=+0.360454368 container init 9891ea5c19e81dcece9932249c20db86ad5fedafdfc74da3c99651ff586872be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-16T14:33:36.255 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:36 smithi026 podman[85446]: 2024-03-16 14:33:36.235890434 +0000 UTC m=+0.363516841 container start 9891ea5c19e81dcece9932249c20db86ad5fedafdfc74da3c99651ff586872be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T14:33:36.532 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:36 smithi026 podman[85446]: 2024-03-16 14:33:36.341338081 +0000 UTC m=+0.468964502 container attach 9891ea5c19e81dcece9932249c20db86ad5fedafdfc74da3c99651ff586872be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T14:33:36.737 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:33:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:33:36.364Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.26:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.26:8765: connect: connection refused" 2024-03-16T14:33:36.737 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:33:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:33:36.365Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.26:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.26:8765: connect: connection refused" 2024-03-16T14:33:36.737 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:33:36 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:33:36.366Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.26:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.26:8765: connect: connection refused" 2024-03-16T14:33:36.872 INFO:journalctl@ceph.osd.2.smithi026.stdout:Mar 16 14:33:36 smithi026 podman[85446]: 2024-03-16 14:33:36.531166435 +0000 UTC m=+0.658792834 container died 9891ea5c19e81dcece9932249c20db86ad5fedafdfc74da3c99651ff586872be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-16T14:33:37.254 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.2.service' 2024-03-16T14:33:37.299 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:37.300 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-16T14:33:37.300 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-16T14:33:37.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.3 2024-03-16T14:33:37.504 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:37 smithi026 systemd[1]: Stopping Ceph osd.3 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:38.004 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:37 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:33:37.604+0000 7f47b1832700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:38.005 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:37 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:33:37.604+0000 7f47b1832700 -1 osd.3 71 *** Got signal Terminated *** 2024-03-16T14:33:38.005 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:37 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:33:37.604+0000 7f47b1832700 -1 osd.3 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:33:40.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:39 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:39.968+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:40.504 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:40 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:33:40.062+0000 7f47ade26700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:12.373296+0000 front 2024-03-16T14:33:12.373314+0000 (oldest deadline 2024-03-16T14:33:39.973898+0000) 2024-03-16T14:33:41.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:40 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:40.859+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:41.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:40 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:40.973+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:41.505 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:41 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:33:41.020+0000 7f47ade26700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:12.373296+0000 front 2024-03-16T14:33:12.373314+0000 (oldest deadline 2024-03-16T14:33:39.973898+0000) 2024-03-16T14:33:42.138 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:41.937+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:42.139 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:41.986+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:42.139 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:41 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:41.822+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:42.255 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:41 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3[66104]: 2024-03-16T14:33:41.972+0000 7f47ade26700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:12.373296+0000 front 2024-03-16T14:33:12.373314+0000 (oldest deadline 2024-03-16T14:33:39.973898+0000) 2024-03-16T14:33:42.950 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:42.777+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:43.236 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:43.168+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:43.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:43.024+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:43.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:42 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:42.948+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:43.505 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:43 smithi026 podman[85551]: 2024-03-16 14:33:43.109487847 +0000 UTC m=+5.602724008 container died 8cfb446f63ee2cb72bc3656dfe435b62e1f4d9f078962977fcbf4eb600d03860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-16T14:33:43.942 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:43 smithi026 podman[85551]: 2024-03-16 14:33:43.706236636 +0000 UTC m=+6.199472796 container cleanup 8cfb446f63ee2cb72bc3656dfe435b62e1f4d9f078962977fcbf4eb600d03860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:43.942 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:43 smithi026 bash[85551]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3 2024-03-16T14:33:44.165 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:43.984+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:44.165 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:43.783+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:44.166 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:43 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:43.950+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:44.255 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:43 smithi026 podman[85579]: 2024-03-16 14:33:43.940777801 +0000 UTC m=+0.826064067 container remove 8cfb446f63ee2cb72bc3656dfe435b62e1f4d9f078962977fcbf4eb600d03860 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:44.486 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:44.163+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:44.578 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:44 smithi026 podman[85670]: 2024-03-16 14:33:44.180155317 +0000 UTC m=+0.017546684 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:33:44.578 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:44 smithi026 podman[85670]: 2024-03-16 14:33:44.283055427 +0000 UTC m=+0.120446779 container create 15fa19511b5d8dc5616c63c865b3115e2a3c80fe457d79bde23797c454043d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T14:33:44.578 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:44 smithi026 podman[85670]: 2024-03-16 14:33:44.502608905 +0000 UTC m=+0.340000265 container init 15fa19511b5d8dc5616c63c865b3115e2a3c80fe457d79bde23797c454043d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c) 2024-03-16T14:33:44.578 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:44 smithi026 podman[85670]: 2024-03-16 14:33:44.506041211 +0000 UTC m=+0.343432570 container start 15fa19511b5d8dc5616c63c865b3115e2a3c80fe457d79bde23797c454043d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T14:33:44.579 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:44 smithi026 podman[85670]: 2024-03-16 14:33:44.576988699 +0000 UTC m=+0.414380060 container attach 15fa19511b5d8dc5616c63c865b3115e2a3c80fe457d79bde23797c454043d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-16T14:33:45.005 INFO:journalctl@ceph.osd.3.smithi026.stdout:Mar 16 14:33:44 smithi026 podman[85670]: 2024-03-16 14:33:44.804812962 +0000 UTC m=+0.642204323 container died 15fa19511b5d8dc5616c63c865b3115e2a3c80fe457d79bde23797c454043d6f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-16T14:33:45.118 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:44.955+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:45.119 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:44.993+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:45.119 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:44 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:44.804+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:45.382 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.3.service' 2024-03-16T14:33:45.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:45.427 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-16T14:33:45.427 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-16T14:33:45.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.4 2024-03-16T14:33:45.458 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:45 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:45.116+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:45.735 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:45 smithi151 systemd[1]: Stopping Ceph osd.4 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:45.986 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:45 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:45.940+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:45.987 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:45 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:45.756+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:45.987 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:45 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:45.733+0000 7f0f22e2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:45.987 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:45 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:45.733+0000 7f0f22e2d700 -1 osd.4 71 *** Got signal Terminated *** 2024-03-16T14:33:45.987 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:45 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:45.733+0000 7f0f22e2d700 -1 osd.4 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:33:46.486 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:46.098+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:46.487 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:46.017+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:47.127 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:46.762+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:47.128 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:46.988+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:47.128 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:46.945+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:47.128 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:46 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:46.945+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:47.486 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:47 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:47.125+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:48.095 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:47 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:47.808+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:48.095 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:47 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:47.808+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:48.096 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:48.034+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:48.096 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:48.034+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:48.096 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:47 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:47.993+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:48.096 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:47 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:47.993+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:48.486 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:48.093+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:48.487 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:48.093+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:22.160647+0000 front 2024-03-16T14:33:22.160634+0000 (oldest deadline 2024-03-16T14:33:48.060530+0000) 2024-03-16T14:33:49.065 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:48.981+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:49.065 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:48.981+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:49.065 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:48.767+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:49.065 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:48 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:48.767+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:49.487 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:49.063+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:49.487 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:49.063+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:49.487 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:49.076+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:49.487 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:49.076+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:22.160647+0000 front 2024-03-16T14:33:22.160634+0000 (oldest deadline 2024-03-16T14:33:48.060530+0000) 2024-03-16T14:33:50.063 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:49.996+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:50.063 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:49.996+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:50.064 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:49.793+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:50.064 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:49 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:49.793+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:50.486 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:50 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:50.061+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.460081+0000 front 2024-03-16T14:33:17.460134+0000 (oldest deadline 2024-03-16T14:33:42.159998+0000) 2024-03-16T14:33:50.487 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:50 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4[35476]: 2024-03-16T14:33:50.061+0000 7f0f1b419700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:22.160647+0000 front 2024-03-16T14:33:22.160634+0000 (oldest deadline 2024-03-16T14:33:48.060530+0000) 2024-03-16T14:33:50.487 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:50 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:50.062+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:50.487 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:50 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:50.062+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:51.017 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:50 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:50.746+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:51.018 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:50 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:50.746+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:51.385 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:51 smithi151 podman[72210]: 2024-03-16 14:33:51.120964459 +0000 UTC m=+5.496609242 container died 89ade8a73db0984fcd8418ab26bd6dfcab59d50e1d4ac2ab52ca9722770fc24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c) 2024-03-16T14:33:51.386 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:51.019+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:51.386 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:51.019+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:51.386 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:51.015+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:51.386 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:51.015+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:51.736 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:51 smithi151 podman[72210]: 2024-03-16 14:33:51.474904607 +0000 UTC m=+5.850549391 container cleanup 89ade8a73db0984fcd8418ab26bd6dfcab59d50e1d4ac2ab52ca9722770fc24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-16T14:33:51.736 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:51 smithi151 bash[72210]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4 2024-03-16T14:33:52.038 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:51 smithi151 podman[72225]: 2024-03-16 14:33:51.750848736 +0000 UTC m=+0.623629577 container remove 89ade8a73db0984fcd8418ab26bd6dfcab59d50e1d4ac2ab52ca9722770fc24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T14:33:52.039 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:51.751+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:52.039 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:51.751+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:52.039 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:51.983+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:52.039 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:51 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:51.983+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:52.336 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:52 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:52.036+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:52.336 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:52 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:52.036+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:52.337 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:52 smithi151 podman[72321]: 2024-03-16 14:33:51.98054948 +0000 UTC m=+0.017801074 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:33:52.337 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:52 smithi151 podman[72321]: 2024-03-16 14:33:52.093710147 +0000 UTC m=+0.130961776 container create 30cd894481806f4a9b8cc992425aaba4c6a45716c7ebc17311752c2e0a7305b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-16T14:33:52.737 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:52 smithi151 podman[72321]: 2024-03-16 14:33:52.437836927 +0000 UTC m=+0.475088512 container init 30cd894481806f4a9b8cc992425aaba4c6a45716c7ebc17311752c2e0a7305b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-16T14:33:52.737 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:52 smithi151 podman[72321]: 2024-03-16 14:33:52.441493973 +0000 UTC m=+0.478745558 container start 30cd894481806f4a9b8cc992425aaba4c6a45716c7ebc17311752c2e0a7305b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-16T14:33:52.737 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:52 smithi151 podman[72321]: 2024-03-16 14:33:52.521099839 +0000 UTC m=+0.558351429 container attach 30cd894481806f4a9b8cc992425aaba4c6a45716c7ebc17311752c2e0a7305b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:33:53.047 INFO:journalctl@ceph.osd.4.smithi151.stdout:Mar 16 14:33:52 smithi151 podman[72321]: 2024-03-16 14:33:52.737263485 +0000 UTC m=+0.774515063 container died 30cd894481806f4a9b8cc992425aaba4c6a45716c7ebc17311752c2e0a7305b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-16T14:33:53.047 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:52 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:52.782+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:53.047 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:52 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:52.782+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:53.047 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:52 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:52.961+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:53.047 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:52 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:52.961+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:53.325 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:53.045+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:53.326 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:53.045+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:53.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.4.service' 2024-03-16T14:33:53.430 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:33:53.430 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-16T14:33:53.430 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-16T14:33:53.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.5 2024-03-16T14:33:53.736 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:53 smithi151 systemd[1]: Stopping Ceph osd.5 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:33:54.022 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:53.940+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:54.022 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:53.940+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:54.023 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:53.808+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:54.023 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:53.808+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:54.023 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:53.768+0000 7f8c2dc0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:33:54.023 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:53.768+0000 7f8c2dc0f700 -1 osd.5 71 *** Got signal Terminated *** 2024-03-16T14:33:54.023 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:53 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:53.768+0000 7f8c2dc0f700 -1 osd.5 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:33:54.023 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:54.020+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:54.023 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:54.020+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:55.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:54.900+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:55.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:54.900+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:55.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:54.977+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:55.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:54.977+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:55.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:54.844+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:55.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:54 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:54.844+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:56.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:56.017+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:56.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:56.017+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:56.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:55 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:55.854+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:56.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:55 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:55.854+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:56.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:55 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:55.873+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:56.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:55 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:55.873+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:56.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:55 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:55.873+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:33:57.236 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:57 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:57.050+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:57.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:57 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:57.050+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:57.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:56.903+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:57.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:56.903+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:57.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:56.882+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:57.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:56.882+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:57.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:56 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:56.882+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:33:58.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:58.058+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:18.083006+0000 front 2024-03-16T14:33:18.082932+0000 (oldest deadline 2024-03-16T14:33:41.582844+0000) 2024-03-16T14:33:58.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5[42830]: 2024-03-16T14:33:58.058+0000 7f8c2a203700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:23.283997+0000 front 2024-03-16T14:33:23.284096+0000 (oldest deadline 2024-03-16T14:33:46.783792+0000) 2024-03-16T14:33:58.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:57 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:57.882+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:58.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:57 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:57.882+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:58.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:57 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:57.849+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:58.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:57 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:57.849+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:58.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:57 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:57.849+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:33:59.237 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:59 smithi151 podman[72426]: 2024-03-16 14:33:59.100193906 +0000 UTC m=+5.452518170 container died 54b538e68a65bed09a3ce42cad0970b60ddcc50e415e21a5f21c6178d9ed6388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T14:33:59.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:58.854+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:59.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:58.854+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:59.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:58.852+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:33:59.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:58.852+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:33:59.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:58 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:58.852+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:33:59.630 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:59 smithi151 podman[72426]: 2024-03-16 14:33:59.410274742 +0000 UTC m=+5.762599006 container cleanup 54b538e68a65bed09a3ce42cad0970b60ddcc50e415e21a5f21c6178d9ed6388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-16T14:33:59.630 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:59 smithi151 bash[72426]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5 2024-03-16T14:33:59.896 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:59 smithi151 podman[72443]: 2024-03-16 14:33:59.628460575 +0000 UTC m=+0.523732489 container remove 54b538e68a65bed09a3ce42cad0970b60ddcc50e415e21a5f21c6178d9ed6388 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-16T14:33:59.897 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:59 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:59.867+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:33:59.897 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:59 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:59.867+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:33:59.897 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:33:59 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:33:59.867+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:32.461923+0000 front 2024-03-16T14:33:32.461909+0000 (oldest deadline 2024-03-16T14:33:58.862270+0000) 2024-03-16T14:34:00.216 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:59 smithi151 podman[72534]: 2024-03-16 14:33:59.92926064 +0000 UTC m=+0.114839352 container create e44b3d5f67ea2fe0f1928557c2d4d080afdc7e2cedeca61f3b5b7bd76155f9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD) 2024-03-16T14:34:00.216 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:33:59 smithi151 podman[72534]: 2024-03-16 14:33:59.833285282 +0000 UTC m=+0.018863925 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:34:00.216 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:34:00 smithi151 podman[72534]: 2024-03-16 14:34:00.140494383 +0000 UTC m=+0.326073043 container init e44b3d5f67ea2fe0f1928557c2d4d080afdc7e2cedeca61f3b5b7bd76155f9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-16T14:34:00.216 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:34:00 smithi151 podman[72534]: 2024-03-16 14:34:00.144021601 +0000 UTC m=+0.329600247 container start e44b3d5f67ea2fe0f1928557c2d4d080afdc7e2cedeca61f3b5b7bd76155f9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-16T14:34:00.217 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:59 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:59.895+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:00.217 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:59 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:59.895+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:00.217 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:59 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:59.895+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:00.217 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:33:59 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:33:59.895+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:00.487 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:34:00 smithi151 podman[72534]: 2024-03-16 14:34:00.215053312 +0000 UTC m=+0.400631971 container attach e44b3d5f67ea2fe0f1928557c2d4d080afdc7e2cedeca61f3b5b7bd76155f9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-16T14:34:00.487 INFO:journalctl@ceph.osd.5.smithi151.stdout:Mar 16 14:34:00 smithi151 podman[72534]: 2024-03-16 14:34:00.437112702 +0000 UTC m=+0.622691359 container died e44b3d5f67ea2fe0f1928557c2d4d080afdc7e2cedeca61f3b5b7bd76155f9b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:34:01.078 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:00 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:00.822+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:34:01.078 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:00 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:00.822+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:34:01.078 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:00 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:00.822+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:32.461923+0000 front 2024-03-16T14:33:32.461909+0000 (oldest deadline 2024-03-16T14:33:58.862270+0000) 2024-03-16T14:34:01.078 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:00 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:00.863+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:01.078 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:00 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:00.863+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:01.078 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:00 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:00.863+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:01.079 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:00 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:00.863+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:01.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.5.service' 2024-03-16T14:34:01.143 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:34:01.143 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-16T14:34:01.144 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-16T14:34:01.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.6 2024-03-16T14:34:01.464 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:01 smithi151 systemd[1]: Stopping Ceph osd.6 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:01.736 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:01.462+0000 7f16ab120700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:34:01.737 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:01.462+0000 7f16ab120700 -1 osd.6 71 *** Got signal Terminated *** 2024-03-16T14:34:01.737 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:01.462+0000 7f16ab120700 -1 osd.6 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:34:02.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:01.845+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:34:02.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:01.845+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:34:02.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:01.845+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:32.461923+0000 front 2024-03-16T14:33:32.461909+0000 (oldest deadline 2024-03-16T14:33:58.862270+0000) 2024-03-16T14:34:02.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:01.913+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:02.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:01.913+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:02.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:01.913+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:02.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:01 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:01.913+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:03.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:02 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:02.875+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:34:03.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:02 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:02.875+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:34:03.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:02 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:02.875+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:32.461923+0000 front 2024-03-16T14:33:32.461909+0000 (oldest deadline 2024-03-16T14:33:58.862270+0000) 2024-03-16T14:34:03.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:02 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:02.886+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:03.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:02 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:02.886+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:03.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:02 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:02.886+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:03.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:02 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:02.886+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:04.236 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:03 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:03.890+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:34:04.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:03 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:03.890+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:34:04.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:03 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:03.890+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:32.461923+0000 front 2024-03-16T14:33:32.461909+0000 (oldest deadline 2024-03-16T14:33:58.862270+0000) 2024-03-16T14:34:04.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:03 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:03.882+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:04.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:03 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:03.882+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:04.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:03 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:03.882+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:04.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:03 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:03.882+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:05.236 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:04.915+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:34:05.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:04.915+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:34:05.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:04.915+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:32.461923+0000 front 2024-03-16T14:33:32.461909+0000 (oldest deadline 2024-03-16T14:33:58.862270+0000) 2024-03-16T14:34:05.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:04.915+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:41.663842+0000 front 2024-03-16T14:33:41.663739+0000 (oldest deadline 2024-03-16T14:34:03.963682+0000) 2024-03-16T14:34:05.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:04.859+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:05.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:04.859+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:05.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:04.859+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:05.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:04 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:04.859+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:06.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:05.940+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:17.359678+0000 front 2024-03-16T14:33:17.359692+0000 (oldest deadline 2024-03-16T14:33:40.859482+0000) 2024-03-16T14:34:06.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:05.940+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:24.260967+0000 front 2024-03-16T14:33:24.261039+0000 (oldest deadline 2024-03-16T14:33:47.160847+0000) 2024-03-16T14:34:06.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:05.940+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:32.461923+0000 front 2024-03-16T14:33:32.461909+0000 (oldest deadline 2024-03-16T14:33:58.862270+0000) 2024-03-16T14:34:06.237 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6[50971]: 2024-03-16T14:34:05.940+0000 7f16a7714700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:41.663842+0000 front 2024-03-16T14:33:41.663739+0000 (oldest deadline 2024-03-16T14:34:03.963682+0000) 2024-03-16T14:34:06.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:05.905+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:06.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:05.905+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:06.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:05.905+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:06.238 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:05 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:05.905+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:07.123 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:06 smithi151 podman[72638]: 2024-03-16 14:34:06.831651067 +0000 UTC m=+5.471342238 container died 15214c61d1d82078ae3ed3f2b869f1f3dd2a65fbbe6e907bf3f1c8b3a1328d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:34:07.124 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:06 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:06.946+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:07.124 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:06 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:06.946+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:07.124 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:06 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:06.946+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:07.124 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:06 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:06.946+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:07.466 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:07 smithi151 podman[72638]: 2024-03-16 14:34:07.221205183 +0000 UTC m=+5.860896354 container cleanup 15214c61d1d82078ae3ed3f2b869f1f3dd2a65fbbe6e907bf3f1c8b3a1328d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307) 2024-03-16T14:34:07.466 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:07 smithi151 bash[72638]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6 2024-03-16T14:34:07.736 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:07 smithi151 podman[72655]: 2024-03-16 14:34:07.463991506 +0000 UTC m=+0.625198158 container remove 15214c61d1d82078ae3ed3f2b869f1f3dd2a65fbbe6e907bf3f1c8b3a1328d26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-16T14:34:08.060 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:07 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:07.934+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:08.060 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:07 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:07.934+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:08.060 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:07 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:07.934+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:08.060 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:07 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:07.934+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:42.139209+0000 front 2024-03-16T14:33:42.139231+0000 (oldest deadline 2024-03-16T14:34:07.439090+0000) 2024-03-16T14:34:08.060 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:07 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:07.934+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:08.061 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:07 smithi151 podman[72749]: 2024-03-16 14:34:07.686148482 +0000 UTC m=+0.018861874 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:34:08.061 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:07 smithi151 podman[72749]: 2024-03-16 14:34:07.797998198 +0000 UTC m=+0.130711577 container create 3f85c18f03046c625387c145c752745ef12f7a26c7e734386079272d2c87d193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-16T14:34:08.061 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:07 smithi151 podman[72749]: 2024-03-16 14:34:07.983645611 +0000 UTC m=+0.316359002 container init 3f85c18f03046c625387c145c752745ef12f7a26c7e734386079272d2c87d193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-16T14:34:08.061 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:07 smithi151 podman[72749]: 2024-03-16 14:34:07.98711876 +0000 UTC m=+0.319832140 container start 3f85c18f03046c625387c145c752745ef12f7a26c7e734386079272d2c87d193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6-deactivate, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-16T14:34:08.061 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:08 smithi151 podman[72749]: 2024-03-16 14:34:08.058675641 +0000 UTC m=+0.391389025 container attach 3f85c18f03046c625387c145c752745ef12f7a26c7e734386079272d2c87d193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-16T14:34:08.486 INFO:journalctl@ceph.osd.6.smithi151.stdout:Mar 16 14:34:08 smithi151 podman[72749]: 2024-03-16 14:34:08.288607844 +0000 UTC m=+0.621321244 container died 3f85c18f03046c625387c145c752745ef12f7a26c7e734386079272d2c87d193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:34:08.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.6.service' 2024-03-16T14:34:08.926 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:34:08.926 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-16T14:34:08.926 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-16T14:34:08.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.7 2024-03-16T14:34:09.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:08 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:08.889+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:09.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:08 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:08.889+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:09.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:08 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:08.889+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:09.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:08 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:08.889+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:42.139209+0000 front 2024-03-16T14:33:42.139231+0000 (oldest deadline 2024-03-16T14:34:07.439090+0000) 2024-03-16T14:34:09.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:08 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:08.889+0000 7fbe97a0e700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24415.0:132 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-03-16T14:34:09.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 systemd[1]: Stopping Ceph osd.7 for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:09.736 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:09.265+0000 7fbe9f422700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:34:09.737 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:09.265+0000 7fbe9f422700 -1 osd.7 71 *** Got signal Terminated *** 2024-03-16T14:34:09.737 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:09.265+0000 7fbe9f422700 -1 osd.7 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-16T14:34:10.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:09.852+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:10.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:09.852+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:10.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:09.852+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:10.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:09 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:09.852+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:42.139209+0000 front 2024-03-16T14:33:42.139231+0000 (oldest deadline 2024-03-16T14:34:07.439090+0000) 2024-03-16T14:34:11.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:10 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:10.867+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:11.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:10 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:10.867+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:11.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:10 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:10.867+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:11.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:10 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:10.867+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:42.139209+0000 front 2024-03-16T14:33:42.139231+0000 (oldest deadline 2024-03-16T14:34:07.439090+0000) 2024-03-16T14:34:12.138 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:11 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:11.827+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:12.138 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:11 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:11.827+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:12.138 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:11 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:11.827+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:12.138 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:11 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:11.827+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:42.139209+0000 front 2024-03-16T14:33:42.139231+0000 (oldest deadline 2024-03-16T14:34:07.439090+0000) 2024-03-16T14:34:13.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:12 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:12.803+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:13.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:12 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:12.803+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:13.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:12 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:12.803+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:13.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:12 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:12.803+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:42.139209+0000 front 2024-03-16T14:33:42.139231+0000 (oldest deadline 2024-03-16T14:34:07.439090+0000) 2024-03-16T14:34:13.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:12 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:12.803+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.151:6804 osd.4 since back 2024-03-16T14:33:47.439475+0000 front 2024-03-16T14:33:47.439521+0000 (oldest deadline 2024-03-16T14:34:12.739422+0000) 2024-03-16T14:34:14.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:13 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:13.790+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6806 osd.0 since back 2024-03-16T14:33:16.834802+0000 front 2024-03-16T14:33:16.834759+0000 (oldest deadline 2024-03-16T14:33:39.734511+0000) 2024-03-16T14:34:14.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:13 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:13.790+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6814 osd.1 since back 2024-03-16T14:33:25.436408+0000 front 2024-03-16T14:33:25.436491+0000 (oldest deadline 2024-03-16T14:33:47.136321+0000) 2024-03-16T14:34:14.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:13 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:13.790+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6822 osd.2 since back 2024-03-16T14:33:33.537750+0000 front 2024-03-16T14:33:33.537764+0000 (oldest deadline 2024-03-16T14:33:55.837675+0000) 2024-03-16T14:34:14.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:13 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:13.790+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.26:6830 osd.3 since back 2024-03-16T14:33:42.139209+0000 front 2024-03-16T14:33:42.139231+0000 (oldest deadline 2024-03-16T14:34:07.439090+0000) 2024-03-16T14:34:14.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:13 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7[59247]: 2024-03-16T14:34:13.790+0000 7fbe97a0e700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.151:6804 osd.4 since back 2024-03-16T14:33:47.439475+0000 front 2024-03-16T14:33:47.439521+0000 (oldest deadline 2024-03-16T14:34:12.739422+0000) 2024-03-16T14:34:14.902 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:14 smithi151 podman[72853]: 2024-03-16 14:34:14.618015479 +0000 UTC m=+5.475598169 container died 384cce9e96b7ed0b79e615bfe77726991f3127843d60aa3741df76aaa083365a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T14:34:15.236 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 podman[72853]: 2024-03-16 14:34:15.081840849 +0000 UTC m=+5.939423532 container cleanup 384cce9e96b7ed0b79e615bfe77726991f3127843d60aa3741df76aaa083365a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:34:15.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 bash[72853]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7 2024-03-16T14:34:15.632 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 podman[72882]: 2024-03-16 14:34:15.332862534 +0000 UTC m=+0.714464499 container remove 384cce9e96b7ed0b79e615bfe77726991f3127843d60aa3741df76aaa083365a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T14:34:15.906 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 podman[72973]: 2024-03-16 14:34:15.534798136 +0000 UTC m=+0.018409126 image pull 3f3b48f8e85f5af956e73d20a98b955dfe8f83b7a1468b1adf419cceff4645e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348 2024-03-16T14:34:15.906 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 podman[72973]: 2024-03-16 14:34:15.650344346 +0000 UTC m=+0.133955332 container create 22eb70058cf5b1109ea722a3b880f196fd39a1a81b6934be3a32f781546e11ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:34:15.906 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 podman[72973]: 2024-03-16 14:34:15.828018092 +0000 UTC m=+0.311629091 container init 22eb70058cf5b1109ea722a3b880f196fd39a1a81b6934be3a32f781546e11ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-16T14:34:15.906 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 podman[72973]: 2024-03-16 14:34:15.83114727 +0000 UTC m=+0.314758259 container start 22eb70058cf5b1109ea722a3b880f196fd39a1a81b6934be3a32f781546e11ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:34:15.906 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:15 smithi151 podman[72973]: 2024-03-16 14:34:15.905096189 +0000 UTC m=+0.388707185 container attach 22eb70058cf5b1109ea722a3b880f196fd39a1a81b6934be3a32f781546e11ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-16T14:34:16.237 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:16 smithi151 podman[72973]: 2024-03-16 14:34:16.127412828 +0000 UTC m=+0.611023819 container died 22eb70058cf5b1109ea722a3b880f196fd39a1a81b6934be3a32f781546e11ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-16T14:34:16.669 INFO:journalctl@ceph.osd.7.smithi151.stdout:Mar 16 14:34:16 smithi151 podman[72999]: 2024-03-16 14:34:16.655825094 +0000 UTC m=+0.523292060 container remove 22eb70058cf5b1109ea722a3b880f196fd39a1a81b6934be3a32f781546e11ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-16T14:34:16.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@osd.7.service' 2024-03-16T14:34:16.713 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:34:16.713 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-16T14:34:16.714 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-16T14:34:16.714 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@rgw.foo.a 2024-03-16T14:34:17.004 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:16 smithi026 systemd[1]: Stopping Ceph rgw.foo.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:17.417 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:17 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a[73595]: 2024-03-16T14:34:17.006+0000 7f499a272700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-16T14:34:17.417 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:17 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a[73595]: 2024-03-16T14:34:17.006+0000 7f49a8af7ac0 -1 shutting down 2024-03-16T14:34:27.300 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 bash[85775]: time="2024-03-16T14:34:27Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-16T14:34:27.301 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 podman[85775]: 2024-03-16 14:34:27.032607944 +0000 UTC m=+10.126541812 container died 3c2ac519e932419586a71b93bfe62b595ad7d854109537a4625a1fd8779cad4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T14:34:27.663 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 podman[85775]: 2024-03-16 14:34:27.472547251 +0000 UTC m=+10.566481160 container cleanup 3c2ac519e932419586a71b93bfe62b595ad7d854109537a4625a1fd8779cad4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-16T14:34:27.663 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 bash[85775]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a 2024-03-16T14:34:27.895 DEBUG:teuthology.orchestra.run.smithi026:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@rgw.foo.a.service' 2024-03-16T14:34:27.920 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 podman[85804]: 2024-03-16 14:34:27.673626311 +0000 UTC m=+0.638499906 container remove 3c2ac519e932419586a71b93bfe62b595ad7d854109537a4625a1fd8779cad4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-16T14:34:27.920 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-16T14:34:27.920 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-16T14:34:27.920 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 systemd[1]: Stopped Ceph rgw.foo.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:34:27.920 INFO:journalctl@ceph.rgw.foo.a.smithi026.stdout:Mar 16 14:34:27 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@rgw.foo.a.service: Consumed 1.623s CPU time. 2024-03-16T14:34:27.934 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:34:27.934 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-16T14:34:27.934 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-16T14:34:27.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@prometheus.a 2024-03-16T14:34:28.237 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:27 smithi151 systemd[1]: Stopping Ceph prometheus.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-16T14:34:28.518 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-16T14:34:28.519 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-16T14:34:28.519 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a[68945]: ts=2024-03-16T14:34:28.241Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-16T14:34:28.519 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 podman[73079]: 2024-03-16 14:34:28.251657792 +0000 UTC m=+0.117981002 container died ab86899811bd9e43122c83c88e07e4e13d2721bb57a1885ce4d0188910254ae2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:28.952 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 podman[73079]: 2024-03-16 14:34:28.708083646 +0000 UTC m=+0.574406854 container cleanup ab86899811bd9e43122c83c88e07e4e13d2721bb57a1885ce4d0188910254ae2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:28.952 INFO:journalctl@ceph.prometheus.a.smithi151.stdout:Mar 16 14:34:28 smithi151 bash[73079]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-prometheus-a 2024-03-16T14:34:29.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@prometheus.a.service' 2024-03-16T14:34:29.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-16T14:34:29.115 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-16T14:34:29.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T14:34:29.313 INFO:teuthology.orchestra.run.smithi026.stdout:Deleting cluster with fsid: efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:34:31.254 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 systemd[1]: Stopping Ceph alertmanager.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:31.505 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a[84560]: ts=2024-03-16T14:34:31.252Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-16T14:34:31.505 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 podman[86283]: 2024-03-16 14:34:31.303852559 +0000 UTC m=+0.148142522 container died ec5ad52188926b68daafe1b225d18cfbc3b48b648184ad3eee6683769f06a7ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:31.796 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 podman[86283]: 2024-03-16 14:34:31.610614586 +0000 UTC m=+0.454904548 container cleanup ec5ad52188926b68daafe1b225d18cfbc3b48b648184ad3eee6683769f06a7ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:31.796 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 bash[86283]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a 2024-03-16T14:34:32.196 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 podman[86297]: 2024-03-16 14:34:31.794902552 +0000 UTC m=+0.484039358 container remove ec5ad52188926b68daafe1b225d18cfbc3b48b648184ad3eee6683769f06a7ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:32.196 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 podman[86297]: 2024-03-16 14:34:31.86207892 +0000 UTC m=+0.551215696 volume remove cb11a67b24c65675074dd856be1f67c2ba8d17d4702b25b6bd207ae56a51c9d7 2024-03-16T14:34:32.196 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-16T14:34:32.196 INFO:journalctl@ceph.alertmanager.a.smithi026.stdout:Mar 16 14:34:31 smithi026 systemd[1]: Stopped Ceph alertmanager.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:34:32.504 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:32 smithi026 systemd[1]: Stopping Ceph node-exporter.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:32.505 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:32 smithi026 podman[86409]: 2024-03-16 14:34:32.433460064 +0000 UTC m=+0.101077725 container died 3ef24dbba39fb474c538e4aa562c830134b41067fb6b4eab89deb3c364501d21 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:32.958 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:32 smithi026 podman[86409]: 2024-03-16 14:34:32.747849721 +0000 UTC m=+0.415467381 container cleanup 3ef24dbba39fb474c538e4aa562c830134b41067fb6b4eab89deb3c364501d21 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:32.959 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:32 smithi026 bash[86409]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a 2024-03-16T14:34:33.254 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:32 smithi026 podman[86423]: 2024-03-16 14:34:32.957547615 +0000 UTC m=+0.529039589 container remove 3ef24dbba39fb474c538e4aa562c830134b41067fb6b4eab89deb3c364501d21 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-16T14:34:33.255 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:32 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-16T14:34:33.255 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:33 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-16T14:34:33.255 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:33 smithi026 systemd[1]: Stopped Ceph node-exporter.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:34:33.255 INFO:journalctl@ceph.node-exporter.a.smithi026.stdout:Mar 16 14:34:33 smithi026 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.a.service: Consumed 1.725s CPU time. 2024-03-16T14:34:33.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-16T14:34:34.041 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:34:35.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:35 smithi151 systemd[1]: Stopping Ceph iscsi.iscsi.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:35.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:35 smithi151 ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a[67363]: debug Shutdown received 2024-03-16T14:34:45.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:45 smithi151 bash[73490]: time="2024-03-16T14:34:45Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-16T14:34:45.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:45 smithi151 podman[73490]: 2024-03-16 14:34:45.627027867 +0000 UTC m=+10.118809847 container died c51e5d771428c960a4ad7b4cff817955dc4ba0f8a1b317975a024eb65a43c24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-16T14:34:46.236 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:46 smithi151 podman[73490]: 2024-03-16 14:34:46.061420582 +0000 UTC m=+10.553202560 container cleanup c51e5d771428c960a4ad7b4cff817955dc4ba0f8a1b317975a024eb65a43c24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T14:34:46.237 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:46 smithi151 bash[73490]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a 2024-03-16T14:34:46.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:46 smithi151 podman[73505]: 2024-03-16 14:34:46.262590649 +0000 UTC m=+0.636159597 container remove c51e5d771428c960a4ad7b4cff817955dc4ba0f8a1b317975a024eb65a43c24f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe6d9f0402dea10394b7e1b4a7b62386c2c108bbf1c5276560690a5cab7d3348, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-yuri6-testing-2024-03-15-0709-5ea4d9c, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-16T14:34:46.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:46 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-16T14:34:46.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:46 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-16T14:34:46.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:46 smithi151 systemd[1]: Stopped Ceph iscsi.iscsi.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:34:46.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi151.stdout:Mar 16 14:34:46 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@iscsi.iscsi.a.service: Consumed 2.166s CPU time. 2024-03-16T14:34:48.367 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:48 smithi151 systemd[1]: Stopping Ceph grafana.a for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:48.630 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:48 smithi151 podman[73776]: 2024-03-16 14:34:48.364045707 +0000 UTC m=+0.122058157 container died 84254fe43d59625b19a891ea74d37ce1efa4b416bf0c5d1b5c11de50934080b0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a, maintainer=Paul Cuzner , version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-type=git, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, release=239.1651231664, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, distribution-scope=public) 2024-03-16T14:34:48.972 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:48 smithi151 podman[73776]: 2024-03-16 14:34:48.7947104 +0000 UTC m=+0.552722850 container cleanup 84254fe43d59625b19a891ea74d37ce1efa4b416bf0c5d1b5c11de50934080b0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., name=ubi8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner ) 2024-03-16T14:34:48.972 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:48 smithi151 bash[73776]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a 2024-03-16T14:34:49.230 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:48 smithi151 podman[73791]: 2024-03-16 14:34:48.970794073 +0000 UTC m=+0.605168195 container remove 84254fe43d59625b19a891ea74d37ce1efa4b416bf0c5d1b5c11de50934080b0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-grafana-a, description=Ceph Grafana Container, vcs-type=git, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-16T14:34:49.230 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:48 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-16T14:34:49.230 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:49 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-16T14:34:49.230 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:49 smithi151 systemd[1]: Stopped Ceph grafana.a for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:34:49.230 INFO:journalctl@ceph.grafana.a.smithi151.stdout:Mar 16 14:34:49 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@grafana.a.service: Consumed 3.409s CPU time. 2024-03-16T14:34:49.486 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:49 smithi151 systemd[1]: Stopping Ceph node-exporter.b for efcc9110-e3a0-11ee-95c9-87774f69a715... 2024-03-16T14:34:49.986 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:49 smithi151 podman[73905]: 2024-03-16 14:34:49.685062556 +0000 UTC m=+0.125510595 container died a06460fc71da51228755ec02c4137ca6e99f039c1f583633a6cd887a16418b56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-16T14:34:50.293 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:49 smithi151 podman[73905]: 2024-03-16 14:34:49.998919839 +0000 UTC m=+0.439367883 container cleanup a06460fc71da51228755ec02c4137ca6e99f039c1f583633a6cd887a16418b56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-16T14:34:50.293 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:49 smithi151 bash[73905]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b 2024-03-16T14:34:50.547 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:50 smithi151 podman[73919]: 2024-03-16 14:34:50.292168997 +0000 UTC m=+0.610751113 container remove a06460fc71da51228755ec02c4137ca6e99f039c1f583633a6cd887a16418b56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-efcc9110-e3a0-11ee-95c9-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-16T14:34:50.547 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:50 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-16T14:34:50.986 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:50 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-16T14:34:50.986 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:50 smithi151 systemd[1]: Stopped Ceph node-exporter.b for efcc9110-e3a0-11ee-95c9-87774f69a715. 2024-03-16T14:34:50.987 INFO:journalctl@ceph.node-exporter.b.smithi151.stdout:Mar 16 14:34:50 smithi151 systemd[1]: ceph-efcc9110-e3a0-11ee-95c9-87774f69a715@node-exporter.b.service: Consumed 1.654s CPU time. 2024-03-16T14:34:51.234 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T14:34:51.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-16T14:34:51.298 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-16T14:34:51.300 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389/remote/smithi026/crash 2024-03-16T14:34:51.300 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/crash -- . 2024-03-16T14:34:51.352 INFO:teuthology.orchestra.run.smithi026.stderr:tar: /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T14:34:51.352 INFO:teuthology.orchestra.run.smithi026.stderr:tar: Error is not recoverable: exiting now 2024-03-16T14:34:51.354 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389/remote/smithi151/crash 2024-03-16T14:34:51.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/crash -- . 2024-03-16T14:34:51.396 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-16T14:34:51.396 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2024-03-16T14:34:51.397 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-16T14:34:51.398 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-16T14:34:51.428 INFO:teuthology.orchestra.run.smithi026.stdout:2024-03-16T14:31:04.656092+0000 mon.a (mon.0) 645 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T14:34:51.428 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-16T14:34:51.428 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[SEC\]' /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-16T14:34:51.498 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[ERR\]' /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-16T14:34:51.572 DEBUG:teuthology.orchestra.run.smithi026:> sudo egrep '\[WRN\]' /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-16T14:34:51.642 INFO:teuthology.orchestra.run.smithi026.stdout:2024-03-16T14:31:04.656092+0000 mon.a (mon.0) 645 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-16T14:34:51.642 INFO:tasks.cephadm:Compressing logs... 2024-03-16T14:34:51.642 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T14:34:51.685 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-16T14:34:51.708 INFO:teuthology.orchestra.run.smithi026.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T14:34:51.708 INFO:teuthology.orchestra.run.smithi026.stderr:: No such file or directory 2024-03-16T14:34:51.708 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-16T14:34:51.710 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T14:34:51.710 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-16T14:34:51.710 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-16T14:34:51.710 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-16T14:34:51.711 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-client.rgw.foo.a.log: /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.1.log: 57.9%gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-16T14:34:51.711 INFO:teuthology.orchestra.run.smithi026.stderr: -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-16T14:34:51.711 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-16T14:34:51.711 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T14:34:51.711 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mgr.y.log: /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.0.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T14:34:51.712 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log 2024-03-16T14:34:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T14:34:51.717 INFO:teuthology.orchestra.run.smithi026.stderr: 93.8% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T14:34:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-16T14:34:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T14:34:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T14:34:51.717 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-16T14:34:51.720 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-16T14:34:51.720 INFO:teuthology.orchestra.run.smithi151.stderr:: No such file or directory 2024-03-16T14:34:51.720 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.6.log: /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.4.log: /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/tcmu-runner.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr: 90.1% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.cephadm.log: 67.8% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/tcmu-runner.log.gz 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr: 86.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr: 81.5% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-16T14:34:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.b.log: 86.1% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-16T14:34:51.722 INFO:teuthology.orchestra.run.smithi151.stderr: 90.5% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-16T14:34:51.730 INFO:teuthology.orchestra.run.smithi151.stderr: 90.5% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T14:34:51.736 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.5% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-16T14:34:51.816 INFO:teuthology.orchestra.run.smithi151.stderr: 92.2% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-16T14:34:51.850 INFO:teuthology.orchestra.run.smithi026.stderr: 88.9% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-16T14:34:51.887 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-16T14:34:51.889 INFO:teuthology.orchestra.run.smithi026.stderr: 92.4% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-16T14:34:51.901 INFO:teuthology.orchestra.run.smithi151.stderr: 93.0% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-16T14:34:51.905 INFO:teuthology.orchestra.run.smithi151.stderr: 93.1% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-16T14:34:51.907 INFO:teuthology.orchestra.run.smithi026.stderr: 93.0% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-16T14:34:51.962 INFO:teuthology.orchestra.run.smithi151.stderr: 93.4% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-16T14:34:51.964 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-03-16T14:34:51.964 INFO:teuthology.orchestra.run.smithi151.stderr:real 0m0.265s 2024-03-16T14:34:51.964 INFO:teuthology.orchestra.run.smithi151.stderr:user 0m0.902s 2024-03-16T14:34:51.964 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m0.061s 2024-03-16T14:34:51.993 INFO:teuthology.orchestra.run.smithi026.stderr: 93.4% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-16T14:34:52.018 INFO:teuthology.orchestra.run.smithi026.stderr: 93.4% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-16T14:34:52.043 INFO:teuthology.orchestra.run.smithi026.stderr: 91.1% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-16T14:34:52.056 INFO:teuthology.orchestra.run.smithi026.stderr: 93.5% -- replaced with /var/log/ceph/efcc9110-e3a0-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-16T14:34:52.058 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-16T14:34:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:real 0m0.362s 2024-03-16T14:34:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:user 0m1.740s 2024-03-16T14:34:52.058 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m0.075s 2024-03-16T14:34:52.059 INFO:tasks.cephadm:Archiving logs... 2024-03-16T14:34:52.059 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389/remote/smithi026/log 2024-03-16T14:34:52.060 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T14:34:52.333 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389/remote/smithi151/log 2024-03-16T14:34:52.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-16T14:34:52.505 INFO:tasks.cephadm:Removing cluster... 2024-03-16T14:34:52.505 DEBUG:teuthology.orchestra.run.smithi026:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 --force 2024-03-16T14:34:52.691 INFO:teuthology.orchestra.run.smithi026.stdout:Deleting cluster with fsid: efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:34:52.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid efcc9110-e3a0-11ee-95c9-87774f69a715 --force 2024-03-16T14:34:52.987 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: efcc9110-e3a0-11ee-95c9-87774f69a715 2024-03-16T14:34:53.105 INFO:tasks.cephadm:Removing cephadm ... 2024-03-16T14:34:53.105 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T14:34:53.121 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-16T14:34:53.135 INFO:tasks.cephadm:Teardown complete 2024-03-16T14:34:53.136 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-16T14:34:53.147 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi026:/dev/vg_nvme/lv_1... 2024-03-16T14:34:53.147 DEBUG:teuthology.orchestra.run.smithi026:> sudo nvme disconnect -n lv_1 2024-03-16T14:34:53.304 INFO:teuthology.orchestra.run.smithi026.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-16T14:34:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:53.306 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi026:/dev/vg_nvme/lv_2... 2024-03-16T14:34:53.306 DEBUG:teuthology.orchestra.run.smithi026:> sudo nvme disconnect -n lv_2 2024-03-16T14:34:53.459 INFO:teuthology.orchestra.run.smithi026.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-16T14:34:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:53.461 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi026:/dev/vg_nvme/lv_3... 2024-03-16T14:34:53.461 DEBUG:teuthology.orchestra.run.smithi026:> sudo nvme disconnect -n lv_3 2024-03-16T14:34:53.594 INFO:teuthology.orchestra.run.smithi026.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-16T14:34:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:53.596 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi026:/dev/vg_nvme/lv_4... 2024-03-16T14:34:53.596 DEBUG:teuthology.orchestra.run.smithi026:> sudo nvme disconnect -n lv_4 2024-03-16T14:34:53.733 INFO:teuthology.orchestra.run.smithi026.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-16T14:34:53.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:53.735 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-16T14:34:53.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/scratch_devs 2024-03-16T14:34:53.762 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_1... 2024-03-16T14:34:53.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_1 2024-03-16T14:34:53.912 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-16T14:34:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:53.914 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_2... 2024-03-16T14:34:53.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_2 2024-03-16T14:34:54.064 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-16T14:34:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:54.066 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_3... 2024-03-16T14:34:54.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_3 2024-03-16T14:34:54.204 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-16T14:34:54.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:54.206 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi151:/dev/vg_nvme/lv_4... 2024-03-16T14:34:54.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo nvme disconnect -n lv_4 2024-03-16T14:34:54.349 INFO:teuthology.orchestra.run.smithi151.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-16T14:34:54.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:54.351 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-03-16T14:34:54.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/scratch_devs 2024-03-16T14:34:54.382 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-16T14:34:54.391 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-16T14:34:54.391 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T14:34:54.394 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-16T14:34:54.408 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-16T14:34:54.412 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T14:34:54.412 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-16T14:34:54.412 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 +936us[ +936us] +/- 99ms 2024-03-16T14:34:54.413 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -465us[ -445us] +/- 69ms 2024-03-16T14:34:54.413 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -46us[ -26us] +/- 62ms 2024-03-16T14:34:54.413 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:34:54.438 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-03-16T14:34:54.442 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-16T14:34:54.443 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-03-16T14:34:54.443 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +933us[ +933us] +/- 99ms 2024-03-16T14:34:54.443 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv02.front.sepia.ceph.com 3 6 377 26 -472us[ -423us] +/- 69ms 2024-03-16T14:34:54.443 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 -55us[ -55us] +/- 62ms 2024-03-16T14:34:54.443 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-16T14:34:54.443 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-16T14:34:54.455 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-16T14:34:54.456 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-16T14:34:54.465 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T14:34:54.545 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-16T14:34:54.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-16T14:34:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:54.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-16T14:34:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:54.683 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-16T14:34:54.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-16T14:34:54.733 INFO:teuthology.task.internal:Duration was 865.427763 seconds 2024-03-16T14:34:54.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-16T14:34:54.782 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-16T14:34:54.782 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T14:34:54.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-16T14:34:54.828 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T14:34:54.828 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-16T14:34:55.291 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-16T14:34:55.291 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-03-16T14:34:55.292 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T14:34:55.313 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-03-16T14:34:55.313 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-16T14:34:55.345 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-16T14:34:55.345 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T14:34:55.356 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-16T14:34:55.414 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-16T14:34:55.414 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T14:34:55.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-16T14:34:55.690 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-16T14:34:55.701 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-16T14:34:55.701 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T14:34:55.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-16T14:34:55.755 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-16T14:34:55.766 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T14:34:55.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-16T14:34:55.793 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-03-16T14:34:55.820 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-03-16T14:34:55.855 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T14:34:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:55.869 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-16T14:34:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-16T14:34:55.909 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-16T14:34:55.921 INFO:teuthology.task.internal:Transferring archived files... 2024-03-16T14:34:55.922 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389/remote/smithi026 2024-03-16T14:34:55.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T14:34:55.962 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389/remote/smithi151 2024-03-16T14:34:55.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-16T14:34:56.000 INFO:teuthology.task.internal:Removing archive directory... 2024-03-16T14:34:56.001 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T14:34:56.003 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-16T14:34:56.049 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-16T14:34:56.061 INFO:teuthology.task.internal:Not uploading archives. 2024-03-16T14:34:56.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-16T14:34:56.072 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-16T14:34:56.072 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T14:34:56.075 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-16T14:34:56.086 INFO:teuthology.orchestra.run.smithi026.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 14:34 /home/ubuntu/cephtest 2024-03-16T14:34:56.103 INFO:teuthology.orchestra.run.smithi151.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 16 14:34 /home/ubuntu/cephtest 2024-03-16T14:34:56.104 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-16T14:34:56.115 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-16T14:34:56.152 INFO:teuthology.nuke:Checking targets against current locks 2024-03-16T14:34:56.181 DEBUG:teuthology.nuke:shortname: smithi026 2024-03-16T14:34:56.181 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T14:34:56.204 DEBUG:teuthology.nuke:shortname: smithi151 2024-03-16T14:34:56.204 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-16T14:34:56.208 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 14:11:21.966966', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9MNqEvWvI8i3vOiK98ta6iTbCewtR19hA3NMWlr1/+lc9ucxHNazUHUXpAoeJ6QHwPZzRJZ6Tc6lHH/K1Zwhk='} 2024-03-16T14:34:56.230 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-15_19:59:43-rados-wip-yuri6-testing-2024-03-15-0709-distro-default-smithi/7603389', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-16 14:11:21.968921', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIbcxlXkGa3pIdZPgFfZ8SvggEytJw1aDR29LjeBlt3FeVccg/EmmW/JWPf+qrViJKt9eFTfKFJlcI+V6VU60JM='} 2024-03-16T14:34:56.261 INFO:teuthology.orchestra.console.smithi026:Power off 2024-03-16T14:34:56.262 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T14:34:56.284 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-03-16T14:34:56.285 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T14:34:56.300 INFO:teuthology.orchestra.console.smithi151:Power off 2024-03-16T14:34:56.300 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-16T14:34:56.309 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-16T14:34:56.322 DEBUG:teuthology.orchestra.console.smithi151:power off output: Chassis Power Control: Down/Off 2024-03-16T14:34:56.322 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T14:34:56.345 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-16T14:35:00.310 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T14:35:00.347 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T14:35:00.420 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-16T14:35:00.457 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is on 2024-03-16T14:35:04.422 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T14:35:04.458 DEBUG:teuthology.orchestra.console.smithi151:pexpect command: ipmitool -H smithi151.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-16T14:35:04.533 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-16T14:35:04.568 DEBUG:teuthology.orchestra.console.smithi151:check power output: Chassis Power is off 2024-03-16T14:35:04.635 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-03-16T14:35:04.669 INFO:teuthology.orchestra.console.smithi151:Power off completed 2024-03-16T14:35:04.773 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} duration: 865.4277634620667 failure_reason: '"2024-03-16T14:31:04.656092+0000 mon.a (mon.0) 645 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-03-16T14:35:04.774 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-16T14:35:04.868 INFO:teuthology.run:FAIL